Android Perfetto Series 10: Binder Scheduling and Lock Contention

Word count: 3.9kReading time: 24 min

 2025/11/16

A long main-thread Binder call can include dispatch delay, server processing, scheduling, and lock waits. This article combines Binder events, thread states, and Java monitor contention to separate those intervals.

These signals must be recorded. Sleeping or a futex stack alone does not establish a Binder or Java-lock bottleneck.

Version note (reviewed October 2026): The original screenshots show different processes and captures, not one complete incident. Binder/ART behavior is checked against android-16.0.0_r1, and SQL/UI against Perfetto v58.3. Recording events, permissions, and vendor behavior remain device-specific. Freezer requirements are distinguished below.

Perfetto Series Catalog
Binder Basics
Perfetto Setup and Data Sources
- Other Binder Analysis Tools
Binder Analysis Workflow
Case Studies
- Case 1: Window Management Delay
- Case 2: Binder Thread Pool Saturation
Platform Features and Best Practices
Summary
References
Attachments
About the Author & Blog

Perfetto Series Catalog

Binder Basics

Binder supports cross-process object calls on Android. This article primarily covers Java/AIDL remote calls through /dev/binder. A local interface in the same process may execute directly without the driver or a Binder worker.

Client: A generated AIDL proxy serializes arguments into a Parcel and submits a transaction through paths such as IBinder.transact().
Server: A Java Stub dispatches through onTransact() to the implementation. Native backends have their own paths, and services need not run in system_server.
Driver: Manages transactions, target buffers, references, and scheduling-related behavior. Different Binder contexts/device nodes are not one communication domain.
Thread pool: Receiving threads can be created on demand. Per-node Oneway serialization and nested synchronous reentrancy also affect dispatch; queuing is not limited to a completely full pool.

Why is Binder needed?

Android commonly isolates apps with processes and UIDs, and one app can use several processes. System capabilities often involve services, while an ordinary local Framework call need not cross a process boundary. Binder standardizes identity, object references, lifetime, and call protocols:

IPC method	Difference from Binder object calls
Socket	Can communicate and expose peer credentials; interface and object lifetime need separate design
Pipe	Provides a byte stream; anonymous pipes often use inherited fds, while named pipes are not limited to parent/child processes
Shared memory	Supports large data and access controls, but requires coordination of synchronization and lifetime

Binder supplies caller identity, but the service still enforces permissions. A caller PID is not always available for asynchronous transactions. Synchronous calls wait for a reply; remote Oneway calls do not wait for server completion, but still incur serialization, driver submission, and possible errors. Priority inheritance depends on the transaction, node, and scheduling policy and does not eliminate lock waits.

The official threading model also describes reentrancy: a thread waiting for a synchronous reply may execute a nested callback. Holding a lock across Binder calls can therefore create deadlocks or reentrant state changes.

Case from App Developer’s Perspective

The screenshot contains AIDL::java::IActivityManager::attachApplication::server. In Android 16, IActivityManager.attachApplication is synchronous. ActivityManager.getService() returns an AIDL proxy wrapping a remote BinderProxy; those are different types. After serialization and submission, the server Stub dispatches to AMS and returns a reply when handling finishes.

Here, app PID 8111 calls thread 2055 in system_server PID 1339. The client binder transaction lasts about 5.255ms, mostly Sleeping but also Running/Runnable. The server interval is not continuously Running either; inspect its thread states separately.

Thread-track binder transaction, binder reply, and flow events establish transaction relationships. AIDL server slices are separate user-space markers; method names may be absent without them. In Perfetto v58.3, aidl_name holds the full marker, while interface holds the interface component. It is not simply aidl_name=IActivityManager.

Synchronous waits often occur through Binder ioctl, but blocked_function reports a kernel blocking function only when the relevant event and symbols are available. It is not a complete Java/native stack, nor is it always epoll_wait or the text ioctl(...).

Perfetto Setup and Data Sources

To diagnose Binder in Perfetto, you need to prepare the data sources and Trace configuration in advance.

Data Sources and Track Overview

Correlate transaction events, scheduler state, and user-space markers:

linux.ftrace: binder_transaction and binder_transaction_received connect send/receive activity for synchronous requests, replies, and asynchronous paths. “Received” is not restricted to the server receiving a request. Sched events add execution and wakeups. binder_transaction_alloc_buf records a successful allocation, not an allocation-failure log.
Atrace categories and app scope: am, wm, and dalvik add Framework/ART markers. App-side markers also need the appropriate atrace_apps scope and device support. dalvik can enable monitor contention but does not guarantee every contention or a full lock-ownership history.
linux.process_stats: Adds process/thread names and mappings. It has recording overhead; a generic “negligible” rating is not a measurement for every device.

android.binder and android.monitor_contention are Trace Processor SQL modules, not additional recording sources. UI names and entry points vary; the v58.3 paths are described below.

Recommended Trace Config

Save this textproto as binder_config.pbtx. The 64MiB ring buffer and 15s duration are example settings; long or busy captures can overwrite early data, so inspect loss statistics. One config cannot guarantee coverage across Android 12–17 and OEM builds.

buffers {
  size_kb: 65536
  fill_policy: RING_BUFFER
}
duration_ms: 15000

data_sources {
  config {
    name: "linux.ftrace"
    ftrace_config {
      ftrace_events: "binder/binder_transaction"
      ftrace_events: "binder/binder_transaction_received"
      ftrace_events: "binder/binder_transaction_alloc_buf"
      ftrace_events: "binder/binder_set_priority"
      ftrace_events: "sched/sched_switch"
      ftrace_events: "sched/sched_waking"
      ftrace_events: "sched/sched_wakeup"
      ftrace_events: "sched/sched_blocked_reason"
      atrace_categories: "binder_driver"
      atrace_categories: "am"
      atrace_categories: "wm"
      atrace_categories: "dalvik"
      symbolize_ksyms: true
      compact_sched { enabled: true }
    }
  }
}
data_sources {
  config {
    name: "linux.process_stats"
    process_stats_config { scan_all_processes_on_start: true }
  }
}

Configuration Item Description

Setting	Evidence	Limit
Binder tracepoints	Transactions and selected driver behavior	Check kernel event availability; missing events do not mean no calls
Sched events	Running/Runnable/Sleep and wakeups	Require sufficiently complete scheduling data; not a business call stack
Atrace am/wm/dalvik	Service semantics and ART contention	Categories, app scope, and build control visibility
Process stats	PID/TID and process/thread mappings	Still distinguish trace-local upid/utid identities

For app markers, add atrace_apps: "actual.package.name" inside ftrace_config. "*" broadens scope and can increase overhead. symbolize_ksyms resolves recorded kernel symbol references; it does not collect a complete kernel call stack. Legacy binder_lock/locked/unlock events are absent from the referenced Linux v6.12 binder_trace.h, so they are omitted rather than assumed available by Android version.

Quick Start: 3 Steps to Capture and View Binder Trace

These paths apply to Android 12+ devices that expose the configuration directory. See Part 2 for permissions and config loading. The output filename is an example; do not overwrite a capture you need.

Start recording with the commands below. The second command records in the foreground. Reproduce on the phone during its 15-second run, not after it returns.

1
2
3

adb push binder_config.pbtx /data/misc/perfetto-configs/binder_config.pbtx
adb shell perfetto --txt -c /data/misc/perfetto-configs/binder_config.pbtx \
    -o /data/misc/perfetto-traces/binder-review.pftrace

Wait for recording to finish, pull the file, and drag it into Perfetto UI:

1	adb pull /data/misc/perfetto-traces/binder-review.pftrace .

Inspect raw tracks and derived views: Find client-thread binder transaction or binder transaction async slices and follow flows to the peer. In v58.3, AndroidBinderViz creates a Binder group with Server Transaction Counts and Client Transaction Counts. The lock plugin registers Android Lock Contention: Toggle View. Names vary by plugin and version. If derived views are absent, use the SQL below to check data coverage.

Other Binder Analysis Tools

Besides Perfetto, two tools are often useful: am trace-ipc (built in) and binder-trace (open source, focused on message decoding).

am trace-ipc: Java Layer Binder Call Tracking

am trace-ipc tracks Java-layer Binder call stacks. In Android 16, AMS requires SET_ACTIVITY_WATCHER and checks eligible processes: either the OS build or the process must be debuggable. A normal user build does not expose arbitrary third-party apps through this command; shell authorization is device-dependent.

Basic usage is very simple, just three steps: “start → operate → stop and export”:

# 1. Start tracking (records Binder calls for eligible processes, usually debuggable ones)
adb shell am trace-ipc start

# 2. Execute the operation you want to analyze on the phone (like starting an app, triggering lag scenarios, etc.)

# 3. Stop tracking and export results to file
adb shell am trace-ipc stop --dump-file /data/local/tmp/ipc-trace.txt

# 4. Pull the result file to computer to view
adb pull /data/local/tmp/ipc-trace.txt

The output aggregates stack counts. This is an illustrative format, not captured output; interface membership and line numbers depend on the target version:

Traces for process: com.example.app
Count: 15
java.lang.Throwable
    at android.os.BinderProxy.transact(BinderProxy.java:xxx)
    at com.example.app.IExampleService$Stub$Proxy.query(...)
    at com.example.app.Caller.query(...)
    ...

It groups recorded stacks by process and occurrence count. It does not report each call’s duration or cover all native/local Binder calls.

Using with Perfetto: Perfetto provides timing, scheduling, and flow context; trace-ipc adds Java caller stacks within its capture scope. Stack collection adds overhead, so distinguish it from normal execution.

Best for: validating frequent IPC as ANR/jank cause, and pinpointing Java-origin call sites.

binder-trace: Real-time Binder Message Parsing

binder-trace uses Frida interception and interface structure definitions to decode messages. The reviewed README at a8a9a06 specifies Python 3.9+, an instrumentation-capable device/emulator, and a matching frida-server. Its bundled Android structure choices are 9–14; -a 11 is not a recipe for Android 16/17.

1 2	# Android 11 example; use the actual device and process name binder-trace -d emulator-5554 -n com.example.app -a 11

-n selects a process name, which need not equal the package name. Consult the tool’s help for pid selection or a matching structure directory. Android/OEM interface differences can affect decoding. Instrumentation adds overhead, so use it for message semantics without treating its timing as an uninstrumented performance measurement.

Binder Analysis Workflow

Do not start by searching the whole trace for anything unusual. Find the transaction you care about, then check its duration, threads, and wait dependencies in that order.

Step 1: Identify Transaction Latency

Define the problem window and client thread, then find transaction slices and flows. Search interface/method names when AIDL markers exist. Sleeping alone is not Binder evidence; correlate it with a transaction or stack.

In Perfetto v58.3 android.binder, synchronous transactions use reconstructed Binder intervals:

client_dur: Wall duration of the client Binder slice, not the entire Java API including all serialization work, and not necessarily all Sleeping.
server_dur: Wall duration of the server binder reply interval, including execution, waits, and nested calls; not pure server CPU time.
dispatch_dur = server_ts - client_ts: Recorded send-to-receive interval, including transfer, queuing, wakeup, and scheduling; not pure thread-pool queue time.
return_dur = client_end - server_end: Remaining time after the server interval ends, including return-path and client scheduling delay.

INCLUDE PERFETTO MODULE android.binder;

SELECT
  binder_txn_id, interface, method_name,
  client_upid, client_utid, client_process, client_thread,
  client_dur / 1e6 AS client_ms,
  server_upid, server_utid, server_process, server_thread,
  server_dur / 1e6 AS server_ms,
  (server_ts - client_ts) / 1e6 AS dispatch_ms,
  ((client_ts + client_dur) - (server_ts + server_dur)) / 1e6
    AS return_ms
FROM android_binder_txns
WHERE is_sync = 1
  AND client_dur >= 0 AND server_dur >= 0
ORDER BY client_dur DESC
LIMIT 20;

The image retains an older query result; the updated query adds instance IDs and return time. Missing names mean no decoded marker, not a nonexistent interface. The module filters incomplete or uncorrelated transactions, so an empty result is not proof that no calls occurred.

When the server interval lies inside the client interval, the three components reconcile with client duration. Investigate negative dispatch/return values as correlation, clock, or data-loss issues rather than clamping them to zero. For dispatch delay examine scheduling, pool availability, and serialization; for server time inspect its work/waits; for return delay inspect client rescheduling.

Step 2: Evaluate Thread Pool and Oneway Queue

A long send-to-receive interval makes worker availability one investigation target. First, a few common questions: how many Binder threads does a process have? What about system_server? When can we call the pool exhausted? Use worker availability and transaction activity to distinguish pool capacity, scheduling, and per-node serialization.

SystemServer’s Binder Thread Pool Scale

In Android 16 ProcessState, mMaxThreads defaults to 15. SystemServer sets sMaxBinderThreads to 31. This configures driver-requested growth; it is not the final total thread count.

startThreadPool() starts an additional thread, and explicit joinThreadPool() callers can add more. That version’s getThreadPoolMaxTotalThreadCount() distinguishes these sources. Apps are not guaranteed exactly 15 workers, nor system_server exactly 31. Java and native Binder do not imply two independent pools of 15 each.

Pool threads normally grow on demand rather than shrinking with momentary load. Check the process instance, Binder context, names, and transaction activity. Counting visible Binder: tracks does not establish the configured cap or that every worker is busy.

Binder Thread Count, Buffer, and “Binder Exhaustion”

When someone says “we ran out of Binder,” do they mean threads, transaction buffers, or object references? Separate those resources before deciding what was exhausted:

Thread pool: Establish that available receiving workers are occupied with transactions/dependencies while new requests are delayed. Exclude Runnable scheduling delay, per-node serialization, and freezing. An S-state worker may be idle or waiting on downstream work; Running/D alone does not prove pool saturation. Synchronous reentrant paths may reuse a thread waiting for a reply.
Transaction buffers: The Android documentation describes about 1MB shared per process, not 1MB dedicated to every call. Android 16 libbinder subtracts two pages from its mapping request; usable space also depends on concurrency, asynchronous quotas, objects, and allocation layout. Allocation failures can return errors rather than merely lengthening a queue. Either request or reply can be too large, and the exception does not establish which business operations already completed.
Objects and references: Retained references consume memory and affect lifetime. There is no universal “Binder object count” equivalent to the 15/31 worker settings. Inspect actual driver/process data before drawing performance conclusions.

For large payloads, consider shared memory, files, or ParcelFileDescriptor with explicit ownership, permissions, and error handling. Bitmap parceling varies by representation and implementation; pixel size alone does not prove fully inline transport. More workers are not a general buffer fix.

Identifying Oneway: A remote call does not wait for a business reply, but the client still serializes, interacts with the driver, and receives submission completion. It is not guaranteed to return instantly without blocking. A local interface does not automatically move work off-thread because it is oneway. Check flags, binder transaction async/binder async rcv, and android_binder_txns.is_sync=0, not a required [oneway] name suffix.

The image shows SystemUI 4024 sending to SurfaceFlinger 1483 with flags 0x11. Duration=0s is an instant event and the flow delay is about 63µs; neither measures the entire Java call or proves all Oneway calls are wait-free. The v58.3 asynchronous table uses receive instants; an available aidl_dur can describe method work. Do not apply synchronous server_dur semantics to it.

Async transactions on one Binder node are serialized, while different nodes may run concurrently. Dense short slices show activity, not queue depth; correlate sends and receives by endpoint with adequate event coverage. Ordinary AMS-to-WMS calls are local, while services can make nested calls to other processes. Follow actual dependencies to determine which clients are affected.

Step 3: Investigate Lock Contention

Interpret S, D, and Runnable intervals separately. D means uninterruptible sleep, not specifically disk I/O. Futex waits can underlie Java monitors, native locks, or condition variables, so the symbol alone does not classify the lock.

Java monitor contention: Recorded ART monitor contention with owner ... slices can be parsed by the v58.3 monitor_contention module. blocked_* identifies the waiter; blocking_* describes the recorded owner. Owner/method data are a snapshot, not proof that one thread held the lock throughout the entire interval.

This image concerns SettingsProvider with PhotonicModulator as the recorded owner, not WindowManagerGlobalLock. Its 164µs contention interval includes roughly 105µs Sleeping, 41µs Running, and 18µs Runnable. Contention duration is neither pure sleep nor the owner’s complete lock-holding duration.

Native/kernel waits: pthread_mutex_lock is a user-space library path, while mutex/rwsem waits can involve kernel locks. sched_blocked_reason supplies a kernel blocking location, not a full native stack. Combine stacks, lock markers, scheduling, and I/O evidence. Native contention is not inherently rarer than Java contention.

SQL for Java Monitor Contention in `system_server` (Optional)

Use the standard library rather than manually splitting slice names. Keep upid/utid to distinguish same-named instances. This lists system_server processes by name while retaining their identities:

INCLUDE PERFETTO MODULE android.monitor_contention;

SELECT
  id, ts, upid, process_name,
  blocked_utid, blocked_thread_name AS waiter_thread,
  blocking_utid, blocking_thread_name AS recorded_owner_thread,
  dur / 1e6 AS contention_ms,
  waiter_count AS recorded_other_waiters,
  short_blocked_method, short_blocking_method,
  blocked_src, blocking_src
FROM android_monitor_contention
WHERE process_name = 'system_server' AND dur >= 0
ORDER BY dur DESC
LIMIT 50;

In Android 16 ART, waiters= records the count before this waiter increments it. Adding one can describe a count including itself at that observation, not a maximum over the interval or a live queue length at query time. The query retains raw waiter_count, leaving unknowns as NULL.

The image uses the older waiter_threads column; do not confuse it with the new raw field. For no results, check dalvik, app scope, build/symbols, event format, and loss. An empty result does not prove no contention.

Case Study: Window Management Delay

These two cases are hypothetical walkthroughs, not measured conclusions from the screenshots above. No matching raw trace is attached; the numbers illustrate accounting.

Suppose a doFrame lasts 30ms in a 60Hz scenario, with 18ms overlapping a synchronous IActivityTaskManager.startActivity transaction. The 16.67ms refresh interval is not automatically the app’s complete deadline; verify FrameTimeline and pipeline timing before inferring presentation from doFrame duration.

Use flow to identify the server thread rather than guessing the interface from Sleeping.
Suppose server duration is 15ms with 12ms of monitor contention. Confirm that contention lies on the transaction’s critical path and the owner is attributable; separate the remaining roughly 3ms of client-side dispatch/return time.
If the owner snapshot points to android.anim, inspect its overlapping work. Name WindowManagerGlobalLock only with supporting stack/log/lock evidence; a method name or contention marker need not identify the lock object’s class.

Candidate improvements include removing redundant startup IPC and shortening confirmed lock-held work. Re-measure the same scenario before claiming a visible-jank improvement.

Case Study: Binder Thread Pool Saturation

Suppose several clients wait on system_server during a burst of IPackageManager queries. A busy-looking Binder track group alone does not establish thread-pool exhaustion:

Confirm that those transactions account for the waits, keeping process instances, nodes, and windows separate.
Check occupied receivers, available workers and growth capacity, plus Runnable delays, freezing, and serialization.
Even if each server call takes about 5ms, whether hundreds of calls saturate the pool depends on arrival rate, concurrency, locks, and CPU load. A large dispatch interval remains a delay to explain.
Attribute slow responsiveness to the burst only when the evidence supports that queue/dependency chain.

Use semantically appropriate batching, deduplication, or rate limits and measure the result. getPackagesForUid returns package names for one UID; it is not a general bulk replacement for PackageManager calls. Service-side rate limits or hotspot optimization may also help.

Platform Features and Best Practices

Cached apps freezer: The official guide dates framework support to Android 11 / API 30. Compatible cgroup/kernel support and device policy are also required; Android 12+ does not universally mean freezing is enabled. A frozen process cannot normally handle transactions. Synchronous calls to it fail and framework checks can terminate the target. Async transactions may be buffered, but capacity is finite and excess traffic can also trigger termination. Cached is not the same as frozen; inspect actual freeze events.

Frozen-callee policy (API 36): RemoteCallbackList.Builder offers DROP, ENQUEUE_MOST_RECENT, and ENQUEUE_ALL. In the Android 16 implementation, use broadcast(Consumer) for these policies; legacy beginBroadcast() throws for non-UNSET policies. ENQUEUE_ALL is bounded (default 1000 in that version, configurable, dropping the oldest callback when full). The implementation falls back without kernel freeze-notification support, so API availability does not guarantee policy-based queuing on every device.

Binder Heavy Hitter Watcher: The Android 16 implementation detects high-frequency call types within batches, grouped by calling UID, Binder class, and transaction code. Frequency hotspots are not necessarily CPU or latency hotspots. Enablement, thresholds, and output are system-configured.

Practical changes:

Use Oneway when the protocol does not require synchronous results, with explicit ordering, backpressure, error, and process-death handling. A keyword change alone does not solve waiting.
Measure Parcel size and concurrent occupancy before choosing shared memory/fds. Do not blindly retry side-effecting transactions after an error.
Keep unpredictable remote work off the UI critical path where the API’s threading contract permits it. Handle lifecycle, stale results, and cancellation; some UI/Framework calls cannot simply move to another thread.

Summary

Start with Binder waits in the problem window, establish transaction relationships, then separate dispatch, server, and return intervals. Explain each using thread states, worker availability, and contention evidence. Preserve uncertainty when events are missing; Sleeping or busy-looking tracks alone do not establish a root cause.

References

Attachments

The previously listed Perfetto/Android_Perfetto_Binder directory was unavailable when checked in October 2026. The current trace repository does not provide a verified matching attachment for these hypothetical cases. Do not treat other traces in that repository as their reproduction data.