1. Overview of Changes (2026-06-04 to 2026-07-17)

Over these six weeks, approximately 150 commits were merged (excluding merge commits), spanning 13 releases (v1.0.29 → v1.1.1). If the main theme of the previous post was “getting all four Agent runtimes working,” this period focused on converging those four runtimes on one shared contract, while continuing to expand the analysis experience, scene knowledge, and public ecosystem.

First, here are several quantifiable changes, calculated with the repository’s documented counting commands:

The sections below follow their impact on everyday analysis workflows, from largest to smallest. We will start with features that changed how SmartPerfetto is used—the dual-trace workspace and Quick Mode—then move to features that raised its capability ceiling, including private analysis context and runtime unification, before covering scenes, knowledge, the ecosystem, and engineering delivery.

2. Dual-Trace Analysis Workspace

Comparison is at the heart of performance optimization: before versus after a change, your product versus a competitor, or regression analysis after a system upgrade. SmartPerfetto already allowed an AI session to attach a reference trace, but the UI exposed only one timeline window. The most natural workflow—viewing two traces side by side—was not possible. The dual-window workspace, shipped on July 10–11 (v1.0.39), filled that gap.

The interaction path is deliberately short:

• Enter with one click: A direct “Open Dual Window” button sits at the top of the AI Panel. Clicking it immediately opens two windows: one showing the current trace and the other initially empty. Select a reference trace from the workspace history without first going through the trace picker.
• Atomic window identity swap: Both windows have trace selectors, so users can freely decide which window displays the current trace and which displays the reference trace. If a historical trace is selected in the current window, the current trace moves atomically to the other window. The workspace never shows the same trace in both windows or enters an intermediate state with no current trace.
• Run identity protection: During analysis, the “current trace + reference trace + Agent session” form a single run identity. Switching traces, exiting comparison mode, changing Providers, or changing workspaces is rejected while the run is active. Only an explicit Stop invokes the backend cancellation protocol. Layout changes, maximization, dragging, and collapsing the AI Panel affect presentation state only: they neither destroy the iframe nor transfer ownership of the streaming connection. This rule ensures that moving a window midway through an analysis cannot destroy an in-progress run.
• Separate exit semantics: Exiting dual-window mode closes only the visual workspace; the reference trace and AI comparison context remain available. Exiting comparison mode additionally clears the reference trace, comparison context, and comparison session.

At the contract level, the smp compare CLI command and the frontend dual-window workspace use the same backend comparison identity, evidence package, and report section. There is no private path that could cause CLI and web comparison results to diverge. The product boundary is equally explicit: comparison always means “the current trace + one reference trace from workspace history.” Comparing any two arbitrary historical traces is not supported; that belongs to the domain of comparing multiple analysis results through snapshot comparison.

For the full interaction model, including its state machine and boundary conditions, see docs/architecture/dual-trace-workspace.md in the repository.

3. Quick Mode: Evidence-Backed Direct Answers and Analysis Receipts

Many everyday questions ask for straightforward facts: “What is the frame rate?”, “Which app is in the foreground?”, or “How severe is the jank?” Sending these questions through a full Agent loop—planning, multiple tool calls, and verification—is both slow and token-intensive. Over these six weeks, the fast path was rebuilt as a genuine evidence-backed direct-answer product. Analysis receipts became a product surface on June 29, the four runtimes were aligned on July 3, and v1.0.38 followed.

The flow has three steps:

1. Intent recognition: A local classifier identifies direct-answer intents, including trace facts such as frame count, duration, processes, and threads; the foreground app; and an initial scrolling-smoothness assessment. It also handles mixed intents. For example, “What is the frame rate? Is it smooth overall?” matches both trace-fact and scrolling-assessment intents and combines the evidence for both in one answer.
2. Deterministic evidence collection: The backend executes the corresponding SQL/Skill directly to collect evidence, bypassing the model’s tool-call loop.
3. Direct-answer assembly: SmartPerfetto assembles the answer directly from evidence tables, adds sentence-level data citations, and returns it after zero model conversation turns. An analysis receipt at the end records which evidence was used and the state of the quality gates.

End-to-end validation covered nine cases against the real DeepSeek API: identity questions, compound facts, global frame count plus jank, initial scrolling assessment, foreground app, the presence of CPU frequency data, questions in English, and more. Eight cases used zero actual model conversation turns, and every case passed claim-by-claim verification.

Two engineering details are worth calling out:

• Consistent behavior across all four runtimes. Direct-answer intent derivation, evidence assembly, and the contract governing the number of conclusions converged on shared modules across Claude, OpenAI, Pi Agent Core, and OpenCode. The Claude path had previously assembled these by hand and silently dropped scrolling evidence for mixed intents, diverging from the other three runtimes. The alignment eliminated this class of drift structurally. Contract values such as the maximum number of conclusions are now injected as template variables from a single source of truth.
• No false activation. Direct answers cover only questions that deterministic evidence can resolve. Questions outside that scope, such as the open-ended request “Give me a quick analysis,” fall back to the normal Agent path instead of being forced into the direct-answer framework and receiving an incomplete answer.

4. Private Analysis Context: Local Source Code + the Android Internals Knowledge Base

A trace can show you where performance degrades, but explaining why often requires returning to source code and system internals. Over these six weeks, SmartPerfetto integrated two types of private context into the analysis pipeline and unified their authorization model in v1.1.0. This was the main reason for the version jump from 1.0 to 1.1.

Local source code (Code-Aware). After a local codebase is registered with SmartPerfetto, AI analysis can map trace behavior to specific classes and functions. The privacy boundary is intentionally strict: the model sees CodeRef metadata—chunk ID, relative path, line range, and symbol name—not raw source text. Raw text is available only through an access-controlled endpoint for the user to inspect. metadata_only and provider_send are separate consent levels; the former guarantees that source snippets are never sent to the model provider. SmartPerfetto also supports patch proposals on top of this model. The AI can propose changes only against source context it has already looked up, and a patch must pass git apply --check before it can be returned with “verified” status.

External Android Internals knowledge base (July 14). A local checkout of android-internals-wiki can be registered as a versioned external knowledge source. The retrieval pipeline enforces the following constraints:

• Paths are denied by default. Only Markdown files under an environment-variable allowlist can be previewed and indexed.
• Registration requires confirmation of content rights (rightsAcknowledged). Sending content to the model provider requires separate, revocable privacy consent (sendToProvider). Once revoked, new indexing and retrieval requests are rejected immediately.
• Only strictly parsed, finalized articles enter retrieval. Drafts and articles awaiting review appear only in the audit trail.
• The model receives only budget-controlled, redacted matching excerpts. Logs and streaming events expose metadata projections only.
• Wiki matches provide system background but cannot replace evidence from the current trace. Diagnostic conclusions must still cite actual SQL/Skill query results.

Combined semantics (v1.1.0). Source code and the knowledge base have independent authorization lists; either can be used alone or both can be combined. Whenever any private context is present, fast/auto mode is forced to resolve to full mode, where the necessary tools and evidence gates are available. Tenant, workspace, user, authorization list, index generation, content fingerprint, and consent state all contribute to a single authorization fingerprint. If any of these changes at a tool-call or run boundary, execution is rejected immediately—fail closed—rather than silently falling back to a different analysis mode. The practical effect is important: after consent is revoked, an index is rebuilt, or permissions change, there is no window in which stale authorization remains active in an existing session.

5. Unifying the Agent Runtime Architecture

This was the largest body of code changed during the six-week period, delivered in two stages.

The early-June refactoring (around v1.0.32, across more than twenty refactoring commits):

• The four runtimes moved into independent engine directories, while shared logic—including diagnostics, hypothesis projection, and runtime descriptors—moved into a common layer.
• Cancellation semantics were extended all the way down to the SQL tool layer. When a user clicks Stop, an in-progress query is actually canceled instead of being allowed to finish before its result is discarded.
• The two third-party runtimes, Pi Agent Core and OpenCode, regained resumability for opaque session state, allowing a restored session to continue with its previous context.
• Provider snapshots are bound to the runtime environment. Restoring a session cannot silently switch models merely because the user has changed the active Provider.
• The verifier’s misdiagnosis-pattern library is loaded from strategy files, so adding a new pattern no longer requires a TypeScript change.

The v1.1.0 consolidation on July 17: All four runtimes now consume the same Provider-independent run specification and comparison-context builder. This structurally eliminates a typical class of bug in which an analysis option takes effect on the Claude path but is silently ignored on the OpenAI path, because options are now parsed in one place. The release also introduced reproducible Perfetto UI prebuild artifacts and a backend regression CI gate (backend-agent-regression-gate in GitHub Actions).

Before v1.1.0 shipped, the project completed two rounds of independent review: a weekly review on July 15 and a clean-room review on July 16, with reports stored under docs/reviews/ in the repository. The release proceeded only after confirming there were no known P0/P1 blockers. v1.1.1, released the same day, fixed more than ten remaining findings from those reviews, including source-reference locatability checks, OpenAI tool-output normalization, verifier evidence-label parsing, and restored scrolling-metric semantics.

6. Token Cost Optimization

In mid-June, we identified the largest source of token consumption in full mode: the methodology section of the system prompt embedded the entire scene strategy—approximately 51K tokens for the launch scene—and the existing prompt-budget mechanism trimmed only supporting sections, never this one. In effect, every full analysis was paying for an entire “textbook on launch analysis.” A complete optimization package landed on June 18–19 in v1.0.35:

• Strategy split: Scene strategies are divided into core and detail layers. The core remains in the prompt and is limited to a few thousand tokens; detail is delivered on demand as the analysis advances through its stages. Quality requirements moved into expected tool-call lists in plan templates and into the verifier, ensuring that a shorter prompt does not produce a shallower analysis.
• Regression baselines using real token counts: New token-baseline tests load the actual strategy files and cover the worst-case combination—scene + architecture + completeness + selection + comparison + source awareness all enabled at once—to prevent future changes from silently inflating prompt size again. The previous unit tests mocked the strategy loader and could not detect size regressions.
• Automatic summarization of large SQL results: SQL outputs above a threshold are summarized automatically, while the raw data becomes a paginated artifact reference without breaking provenance.
• Compressed tool descriptions and cache boundaries: MCP tool descriptions were compressed across all four runtimes. Prompt-cache boundaries were added to the runtimes and are now selected according to capability flags declared by Provider adapters, rather than hardcoded Provider names.

The benefit depends on Provider capabilities. For Providers without prompt caching—OpenAI-compatible endpoints, Pi, and OpenCode—the optimization directly reduces the cost of every request. For Providers with caching, it primarily lowers cache-creation cost and context-window pressure.

7. Expanded Scene Coverage: Camera, Heap, and GPU

All three new scenes follow the same principle: evidence first; when data is insufficient, say so explicitly instead of inventing a conclusion.

• Camera analysis (July 12): SmartPerfetto added an evidence-oriented camera capture preset, available directly through smp capture, along with a camera-trace evidence Skill. The key design is conditional tracepoint availability. Tracepoints for different stages of the camera pipeline vary significantly across vendors and platform versions. The Skill first probes the current trace to determine which evidence sources exist, reaches conclusions only for stages backed by data, and explicitly marks data-starved stages as unavailable instead of forcing them through a template.
• Heap dominator-path analysis (July 14): A new deterministic Skill clusters and extracts heap dominator paths. It extracts dominator paths from the managed heap graph, groups isomorphic paths into clusters, and answers, “Which reference chains are retaining this memory?” Batch trace runs automatically aggregate heap paths across traces and present the cross-trace distribution of memory clusters in the batch report, backed by regression cases using real heap-graph traces.
• GPU compute evidence (July 14): A vendor-neutral GPU compute evidence Skill that does not depend on proprietary counters from any specific GPU vendor, backed by real-trace regression coverage.
• Existing-scene hardening (early June): Memory-analysis evidence handling was hardened, and the evidence semantics for thread blocked reasons were corrected. The latter fixed incorrect attribution in the “blocked reason” field.

8. RFC-0025: Eight AI Workflow Surfaces

Eight capabilities defined by RFC-0025 were merged on July 7. The goal was to extend AI analysis from a question-and-answer interaction into auditable, batch-oriented, collaborative workflows. Design and implementation documents live under docs/architecture/perfetto-ai-rfc-0025/ in the repository. Before integration, each capability went through a complete independent review, and every P1 finding was fixed.

9. Knowledge Foundation: Android 17 Rendering Pipelines and the Trace Case Corpus

• Android 17 rendering-pipeline knowledge sync (July 13): The authoritative S01–S14 content under docs/rendering_pipelines/ is now synchronized from a source repository pinned to a specific commit (npm run sync:rendering-pipelines). Manual edits are prohibited, preventing the knowledge base from drifting from its source. These are not ordinary documentation files: rendering-pipeline detection, teaching mode, and some Skills read them directly at runtime through doc_path. They are therefore bundled into every distribution format, including the Docker image, whose previously missing copy was fixed as part of this work.
• Trace case corpus (July 13–14): SmartPerfetto introduced the Trace/catalog.json corpus catalog contract and migrated all real traces into a directory-structured corpus. It also introduced deterministically constructed traces: scenarios such as binder I/O blocking, GPU load, and framework pipelines are represented as reproducibly generated synthetic traces, each with a file of expected conclusions. Scene regression coverage expanded from “six fixed traces” to “catalog-driven corpus regression.” Adding a new scene now requires only registering its case in the catalog, without changing the regression framework.

10. The Case Knowledge Platform and Self-Evolution V1

The first version of the Case Knowledge platform landed on June 20–22 (v1.0.36/37), with the goal of making analysis experience persistent and reusable:

• Curated case ingestion: Typical performance-problem cases can be organized into structured knowledge and retrieved for citation during analysis.
• Self-Evolution V1: Patterns confirmed during analysis can be captured as reusable knowledge entries. Hardening this pipeline accounted for most of the work: a research report and multiple rounds of independent review drove fixes for 14 gaps, including automatic confirmation scans for patterns, serialized writes to pattern storage so concurrent analyses cannot overwrite one another, and SQL-result persistence.
• Runtime memory-layer hardening: SmartPerfetto now records previously silent state such as Provider continuity breaks and CLI fallback sources. When something goes wrong, users can reconstruct which model was actually used and which fallback path the run took.

11. Ecosystem: Eight Repositories Form a Complete Path

SmartPerfetto is no longer an isolated project. Launched on July 14, the Android Performance Ecosystem connects eight repositories across instrumentation, capture, analysis, knowledge, and reproduction into a complete Android performance analysis path:

The relationship between SmartPerfetto and Perfetto-Skills deserves a closer look. The two repositories evolve independently but share a portable analysis contract:

• Public Skill export system (July 11-12): backend/skills/public-export.yaml explicitly classifies every runtime asset. Portable workflows, SQL, methodology, and pipeline knowledge enter the public projection, while product-specific Provider, session, and streaming semantics remain in SmartPerfetto. npm run verify:public-skills verifies the exported result.
• Bidirectional impact review gate: npm run check:perfetto-skills-impact identifies every commit that touches a shared contract—including Skills, strategies, evidence contracts, SQL services, pipeline knowledge, and export policies—and requires the author to explicitly record a coordination decision: sync required, sync not required, or deferred with a handoff link. Contract drift no longer depends on human memory.
• Perfetto upstream syncs: This period included three upstream syncs (v56.0 → v57.1 → v57.2). Each followed the repository’s sync rules for merging the fork, refreshing the UI prebuild, and running scene regressions.

12. Engineering and Usability

• Real-path DeepSeek end-to-end verification: Agent SSE end-to-end tests now use the real DeepSeek API by default (npm run verify:e2e:deepseek*), with a manually triggered GitHub Actions workflow also available. Only a real Provider path can expose behavioral differences in tool-call formats and streaming chunks.
• Institutionalized review: Weekly architecture reviews are paired with clean-room reviews before releases, with reports committed under docs/reviews/. Quick Mode, RFC-0025, and v1.1.0 all had independent review records before these major features shipped.
• Usability fixes: Portable packages exclude runtime data (v1.0.33); macOS signing resources are staged correctly (v1.0.37); the launcher has a port fallback; Bedrock model IDs are normalized across regions; the OpenCode runtime is linked correctly inside Docker; and Agent turn budgets have been adjusted.
• Dependency maintenance: Core dependencies—including the Claude Agent SDK, OpenAI Agents, Pi Agent Core, and OpenCode SDK—have been upgraded continuously.

13. The Complete SmartPerfetto Feature Map

As of v1.1.1, SmartPerfetto’s complete feature map is organized into eight layers: “entry points → analysis → evidence and quality → knowledge → comparison and code awareness → output → extensibility and governance → distribution.”

14. Release Timeline

15. Try It and Send Feedback

Choose the distribution format that fits your use case (see the repository README and docs/getting-started/quick-start.md for details):

• For the fastest trial: Use the Docker Hub image with docker compose -f docker-compose.hub.yml up -d; no host installation of Node.js is required.
• If you do not want Docker: Use a GitHub Release portable package for Windows, macOS, or Linux; it includes every runtime.
• For terminal and scripting workflows: Run npm install -g @gracker/smartperfetto; Node.js 24 is required on the host.
• If you want to modify the code: Check out the source and run ./start.sh; use ./scripts/start-dev.sh for plugin UI development.

When reporting a bug, please include as much of the following as possible: a reproducible trace, the exact evidence location (timestamp/process/thread), and the difference between the expected and actual analysis conclusions. Issues that include a trace receive the highest priority.

The previous SmartPerfetto update was published on May 17. At that point the project had already moved from “an AI Assistant inside Perfetto UI” to “a reusable trace analysis platform.” By June 4, the new work is concentrated in five areas: Smart Mode, selected-range quick analysis, CLI capture, stronger evidence rules for Power / ANR / Input / IO / Network, and four Agent runtimes.

This article is based on the SmartPerfetto main branch on June 4, 2026. The latest public release at this point is v1.0.28. The goal is simple: list what changed after May 17, explain how runtimes and evidence sources are handled, and show what information makes a useful bug report.

Project links:

• SmartPerfetto main repository: https://github.com/Gracker/SmartPerfetto
• Previous update: https://www.androidperformance.com/en/2026/05/17/SmartPerfetto-Two-Week-Update/

2026-05-17 to 2026-06-04 Overview

The feature map in the previous article still showed SmartPerfetto v1.0.7. The public release is now v1.0.28. At this commit, the repository has 230 *.skill.yaml files. After excluding _template, 226 of them are functional Skill files. backend/strategies/ has 20 .strategy.md files and 35 .template.md files.

The count changed, but the useful change is not the count itself. Users will notice the changes in six areas:

The main direction is clear: more scenes now have verifiable analysis flows. The model still explains and prioritizes. The numbers still come from trace_processor_shell, SQL, Skill tables, and structured evidence in reports.

Current Feature Map

SmartPerfetto now has five practical layers: entry, analysis, evidence, runtime, and distribution.

One rule matters before everything else: SmartPerfetto does not push the whole trace into the prompt. The model sees SQL results returned by backend tools, Skill output, scene context, report summaries, and structured evidence. New runtimes do not bypass that rule.

Feature One: Smart Mode, Selected-Range Quick Analysis, and CLI Share One Core

The previous article already covered fast / full / auto analysis modes. After May 17, SmartPerfetto added two more practical entries: Smart scene selection and range-scoped quick analysis.

Smart Mode starts with scene inventory. A mixed trace can contain startup, scrolling, clicks, navigation, ANR, and device-state changes in the same file. SmartPerfetto first identifies analyzable ranges, attaches eligibility, confidence, context, verification, and report IDs, then lets the user deep-dive all scenes or only one scene type.

Selected-range quick analysis solves a smaller but common problem. In Perfetto UI, users often select a time range or one track event and ask “why is this range slow?” or “what happened around this slice?” The backend now passes that selected context into the request. The quick path gathers evidence around that range instead of treating the whole trace as the analysis target.

The CLI entry also became a real terminal product instead of a thin backend wrapper. smp capture handles capture, smp analyze handles analysis, and compare workflows, session files, report output, and Provider/runtime selection reuse the Web logic. CLI does not start the Web UI, and it still uses the shared analysis core.

This is the reason the four-runtime work can fit into the product. The entry may be UI, CLI, or API, but the implementation returns to AgentAnalyzeSessionService, Skills, MCP tools, and report checks.

Feature Two: Evidence Rules Now Cover More Scenes

Earlier SmartPerfetto work was heavily focused on startup and scrolling. The late-May work expands the rule set for what a report must prove in more scenes.

This is bigger than adding a few SQL queries. Before, report quality relied mostly on prompts asking the model to cite evidence. Now strategy frontmatter, Skill evidence rules, final report checks, and claim verification work together. When key numbers, threads, processes, frame counts, or percentages appear in a report, they need to point back to their evidence source.

That is also why failed SQL diagnostics cannot become frontend evidence. Failed SQL is useful for maintainers. It is not a trace fact for the user. Verified data, diagnostics, visible report text, and snapshot provenance are stored separately.

Feature Three: Four Runtimes Share One Product Output

The previous article covered Claude Agent SDK and OpenAI Agents SDK. In v1.0.28, SmartPerfetto has four runtimes:

Provider Manager and env config do not guess the runtime from whichever key exists. The priority is request/session providerId, active Provider Manager profile, SMARTPERFETTO_AGENT_RUNTIME, then the default claude-agent-sdk. Once an analysis session is created, it pins the credential source used at that moment. Restoring an old session does not silently switch to a new active provider.

The internal mechanisms differ. Claude runtime exposes an in-process MCP server. OpenAI runtime adapts the shared tool descriptors into function tools. Pi Agent Core uses request-scoped native tools. OpenCode calls SmartPerfetto tools through an isolated server and a request-scoped MCP bridge.

The product output is shared:

1
2
3
4
5
6

runtime result
  -> agentResultNormalizer
  -> finalReportContractGate
  -> evidence rules / claim verification / identity resolutions
  -> HTML report / CLI turn files / analysis-result snapshot
  -> frontend visible projection

The release checks follow that output format. Before v1.0.28, Claude, OpenAI, Pi Agent Core, and OpenCode all passed startup / scrolling strict E2E. All 8 E2E runs required claim verification to pass, 12 checked claims, 0 unsupported / issues, and a non-partial final report. The release flow also ran 6 canonical scene trace regressions, frontend prebuild checks, CLI pack checks, npm fresh-install smoke, portable packaging, GitHub Release checks, and Docker Hub manifest checks.

This does not mean every runtime prints identical wording. Different SDKs have different tool-call timing, streaming events, context recovery, cost, and timeout behavior. SmartPerfetto constrains the shared surfaces: SSE events, HTML reports, CLI turn files, analysis-result snapshots, claim verification, and session resume semantics.

Feature Four: Reports and Debugging Data Are Easier to Inspect

Some changes are less visible than runtime support, but they matter more in daily use.

The first group is report completeness. Changes such as ensure complete agent final reports, final report presentation hardening, evidence table readability, and analysis quality rules all address one problem: running many tools does not guarantee a complete user-facing report. A report now needs to satisfy the final report rules defined by the scene, and its claims need to pass claim verification. The frontend may clean up visible text. Provenance needed by reports, CLI, or snapshots must stay intact.

The second group is SQL and diagnostics. Raw SQL artifacts, failed SQL diagnostics, and trace processor startup failures are handled more explicitly. Failed SQL diagnostics can go into logs and debugging material, but they should not be mixed into the evidence table for the current trace. If trace_processor_shell fails to start, a prebuilt binary is missing, or trace loading fails, the error should show up in health checks or error output, not as a vague AI-quality problem.

The third group is codebase-aware analysis. SmartPerfetto can include local codebase context in an analysis task. This helps map trace evidence to code. It does not replace trace evidence. Code locations can explain possible fixes; performance judgment still comes from trace data.

This makes bug reports more useful. Instead of only saying “the AI conclusion is wrong,” users can attach a Result ID, HTML report, evidenceRefId, claim ref, related SQL, Skill table, and /health runtime data.

Usability Fixes

Recent fixes also focus on whether a real user can run the first trace and diagnose a failure.

These fixes are not flashy. They decide whether Android performance teams can use the tool on real traces. SmartPerfetto crosses Perfetto UI, Express backend, SSE, trace processor, Provider protocol, model SDK, CLI, Docker, and portable packages. State drift in any of these surfaces usually shows up to the user as “the AI analysis is unreliable.”

How to Choose a Run Path

For most users, Docker or the portable package is still the first choice. Developers and script users can choose source or CLI.

For runtime config, pick one mode first:

Do not put provider names into SMARTPERFETTO_AGENT_RUNTIME. That variable only accepts claude-agent-sdk, openai-agents-sdk, pi-agent-core, and opencode. Providers such as DeepSeek, Qwen, Kimi, MiMo, and TokenHub should be configured through Provider Manager or the matching protocol env block.

How to File a Useful Bug Report

SmartPerfetto bug reports still belong in GitHub Issues. Security reports should not be filed as public issues; use GitHub private advisory or the security contact instead.

After v1.0.28, a useful report should include:

Provider issues should separate “the model can chat” from “the model can run stable tool / function calls.” SmartPerfetto depends on streaming output, structured tool arguments, context recovery, and long report generation. Normal chat success does not prove full trace analysis works.

Trace issues also need capture conditions. Android 12+ traces with FrameTimeline are better for startup, scrolling, and rendering analysis. Power, ANR, network, and IO each need different categories. When the data source is missing, the correct behavior is to report missing signals, not to invent a confident conclusion.

Feedback That Helps Most Right Now

SmartPerfetto needs feedback that can become a code fix, a Skill branch, or a regression test.

Useful samples look like this:

• A power trace where the report blames CPU, but manual inspection points to wakelock or modem activity. Attach power rails, wakelock, CPU freq / idle evidence.
• An ANR trace where the report says the main thread is blocked by futex, but logcat or thread state does not support it. Attach the ANR range, main-thread state table, futex wait table, and logcat snippet.
• An OpenAI-compatible provider passes connection testing but mutates tool arguments during full analysis. Attach runtime, provider, model ID, protocol type, session log, and /health.
• An OpenCode or Pi Agent Core config passes smoke but startup / scrolling E2E quality is unstable. Attach model config, command, Result ID, and HTML report.
• Smart scene detection picks the wrong range. Attach the trace range, scene inventory output, expected scene type, and manual evidence.
• CLI capture or analyze fails on a remote machine. Attach Node version, platform, smp doctor --format json, CLI turn files, and trace processor error.

A screenshot plus “the result is wrong” is hard to act on. The maintainer needs to know whether the problem is trace capture, SQL, Skill logic, scene strategy, Provider protocol, runtime session, frontend display, or report rules.

Closing

Between May 17 and June 4, SmartPerfetto moved from answering trace questions inside Perfetto UI toward delivering trace-analysis results through multiple entries, multiple runtimes, and shared evidence rules.

It is still moving quickly. Provider presets, runtime config, public APIs, report rules, and the Skill system will continue to change. For ordinary users, the best next step is to run Docker or a portable package on a real trace. For contributors, the most useful inputs are reproducible traces, wrong-attribution reports with evidence pointers, SQL that can improve a Skill or strategy, and regression samples.

The May 17 article no longer describes the current repository state. If you try SmartPerfetto again now, start with Smart Mode, selected-range quick analysis, CLI, and the four runtime options.

When I wrote the SmartPerfetto open-source introduction on April 29, the headline was still “put an AI Assistant that can run SQL, invoke Skills, and generate reports inside the Perfetto UI.” Two weeks later the repository has moved a long way. The feature surface expanded from single-trace Q&A to reusable analysis results, multi-trace comparison, dual Claude/OpenAI runtimes, SQL guardrails, evidence-source indexing, no-install packages, rendering-pipeline teaching, and a more complete Provider diagnostic flow.

This article is based on the SmartPerfetto repository state as of May 17, 2026, and adds a new feature description on top of the previous post. Readers should walk away knowing three things: what was added in the past two weeks, where the current full feature boundary sits, and what information a good bug report should include.

Project links:

• SmartPerfetto main repository: https://github.com/Gracker/SmartPerfetto
• Previous open-source introduction: https://www.androidperformance.com/en/2026/04/29/SmartPerfetto-Open-Source-Perfetto-AI-Assistant/

What Changed Most in Two Weeks

Several numbers from the previous article are already outdated. Back then the repository had 165 YAML Skills and 12 scene strategies. In the current repository state, backend/skills/ has 220 *.skill.yaml files, and after excluding _template that comes to 216 functional Skill files. backend/strategies/ has 18 .strategy.md files and 26 .template.md files.

This is not a matter of piling files on top of files. The changes land in six areas that users will actually feel:

The past two weeks also fixed a batch of usability edge cases. The large-trace upload limit moved up to 5 GiB with admission and disk pre-checks; the startup script now cleans orphan trace_processor_shell processes and opens the browser automatically; Linux glibc issues, IPv6-disabled hosts, Claude SDK native binary selection, Docker env provider priority, OpenAI/MiMo tool calling compatibility, and Perfetto frontend prebuilt asset verification all received fixes.

These fixes do not look like new features, but they decide whether a real user can finish analyzing the first real trace.

Current Full Feature Map

SmartPerfetto now reads as five layers: entry, analysis, evidence, output, and runtime.

The most important thing to underline here is the evidence boundary. SmartPerfetto does not push the entire trace file into the prompt. The model only sees SQL results returned by backend tools, Skill output, report summaries, and structured context. Performance numbers still come from trace_processor_shell and deterministic computation inside Skills.

New Feature One: Multi-Trace Analysis Result Comparison

The older trace comparison is closer to “current window + reference trace” real-time analysis. That mode is still preserved and works well when you want the AI to query two raw traces in the same conversation.

The new multi-trace analysis-result comparison swaps the comparison subject. It compares completed analysis results. After every AI analysis finishes, SmartPerfetto saves key metrics, evidence references, the conclusion contract, and the report entry into an analysis result snapshot. When the UI shows Ready result or Partial result at the top, it means this analysis now has a reusable result.

There are two entry points:

• Type something like compare with another one, compare AR-1234abcd, or compare AR-11111111 and AR-22222222 into the AI input box.
• Click the fact_check icon at the top to open the result picker and manually choose the baseline and one or more candidates.

The problem this solves is not opening two traces at the same time; it is reusing completed analysis results. It does not require another Perfetto UI window to stay open, and it supports sharing within a workspace. Results are private by default; once a user explicitly shares one, users in the same workspace can include it as a candidate result for comparison.

Output happens in two layers:

• The chat message gives the Comparison ID, baseline/candidates, the number of significant changes, and the first few rows of key metrics.
• The HTML comparison report expands the standard metrics, the delta, the input snapshots, the significant changes, and the AI conclusion.

The currently standardized metrics focus on comparable items such as startup duration, FPS, Jank, and slow frames. When metrics are missing, the backend tries to backfill them. If the backfill fails, the comparison still completes but the missing reason is shown.

This is useful for performance regression work. A startup optimization, a scrolling feel adjustment, or a cross-device re-test can each finish AI analysis on its own trace and then sit inside the same comparison report so the differences are visible.

New Feature Two: SQL, Evidence Sources, and Upstream Diagnostic Capabilities

The part of the recent work that deserves the most explanation is how SmartPerfetto handles evidence sources. What the user sees is still conversation, tables, and reports. Internally there are three new layers of contract: how SQL becomes the final executable version, which tool call produced each table/summary, and which exact evidence row a number in the conclusion came from.

The first layer is the SQL guardrail. execute_sql, execute_sql_on, raw SQL auto include, and the Skill validator now share the same stdlib dependency analyzer that identifies tables, functions, macros, and INCLUDE PERFETTO MODULE directives inside SQL. This brings several direct benefits:

• When raw SQL references a Perfetto stdlib table or function, the backend does its best to auto-complete the required module.
• The Skill validator checks whether the prerequisites.modules declared in a Skill cover the stdlib tables/functions/macros actually used by the SQL.
• The guardrail checks low-noise SQL risks such as repeatable CREATE, safe SPAN_JOIN usage, and direct parsing of the args table.
• When the AI Assistant displays and copies SQL, it uses the final executable SQL, run through the syntaqlite formatter. If the formatter fails, it falls back to the original SQL without blocking the analysis.

The boundary should be spelled out: the current guardrail is an engineering contract, not a full Perfetto SQL AST parser. It reduces frequent misjudgment but does not prove that every future Perfetto SQL syntax is statically covered.

The second layer is the stdlib docs and lineage knowledge base. The repository adds backend/data/perfettoSqlDocs.json, generated from upstream stdlib_docs.json, with module, entity, direct/transitive includes, and optional pfsql lineage status. lookup_sql_schema, list_stdlib_modules, and query_perfetto_source can return the official stdlib documentation, include relationships, and source-code search results to the Agent together. The model no longer needs to rely only on memory when writing SQL, and we do not need to hard-code every stdlib detail into the prompt.

The third layer is the data-source index. DataEnvelopes produced by SQL and Skills can carry evidenceRefId, traceSide, traceId, queryHash, sourceToolCallId, paramsHash, plan phase, toolNarration, and producerReason. The HTML report and the analysis result snapshot preserve these fields. The current report contract also adds “sentence-level data citations (structured sources)”: when key numbers, thread names, process names, frame counts, or percentages appear in the conclusion, they can be mapped to evidence_ref_id, source_ref, row_index / row_selector, column, and value.

Quick answers also use the same format. Internally we first derive the conclusion contract from the raw conclusion text, keep machine-readable evidence IDs, and then produce a cleaned-up version for the user. The analysis result snapshot persists this contract so that, when used later for multi-trace comparison or report review, the source references are not lost just because the frontend cleaned up the display.

This matters for debugging. Users do not have to just say “the AI conclusion is wrong”; they can point to Q1, a specific evidenceRefId, a specific Result ID, or a specific column of a specific table. Maintainers can then judge whether the problem lives in SQL, the Skill, the trace data source, the model summary, or report rendering.

This round also productized a batch of upstream Perfetto diagnostic experience into YAML Skills:

These capabilities do not change SmartPerfetto’s core principle: the model explains and prioritizes, but the numbers still come from trace_processor_shell, SQL, Skills, and traceable data sources in reports.

New Feature Three: Provider Manager and Dual Runtimes

Model configuration in SmartPerfetto now has three boundaries:

• Connection configures the SmartPerfetto backend address and an optional SMARTPERFETTO_API_KEY. It is not a model-vendor key.
• Providers configures model provider profiles, including Base URL, API key/token, model ID, and protocol type.
• The runtime decides which SDK the backend uses to orchestrate tool calls: claude-agent-sdk or openai-agents-sdk.

Runtime selection priority is:

Claude Agent SDK fits Anthropic, Claude Code local auth, Bedrock, Vertex, and Anthropic/Claude-compatible providers. OpenAI Agents SDK fits the OpenAI Responses API, Ollama, and OpenAI-compatible gateways that support streaming tool/function calling.

Provider Manager profiles take precedence over .env fallback. An analysis session that has already been created pins the credential source at that moment. When that session is restored, it does not switch to a different model just because the active provider was changed afterwards. This constraint is important because multi-turn follow-up questions in trace analysis depend on the previous turn’s tool results and SDK session state.

When debugging, do not just look at .env. Open http://localhost:3000/health and inspect aiEngine.runtime, aiEngine.credentialSource, aiEngine.providerMode, and the diagnostics. Many provider presets come from public documentation and may change with account region, plan, or console domain. When connection fails, streaming output behaves oddly, or tool calls are unstable, check the Base URL, model ID, and protocol type in the provider’s own console first.

New Feature Four: Rendering Pipeline Teaching Moves From Static Docs to the Current Trace

The “rendering teaching” surface has been refactored over the past two weeks. The previous version was closer to “detect one main pipeline and show the matching Mermaid, the key slices, and the teaching doc.” The new goal is different: show what really happened between App, Framework, RenderThread, Producer, BufferQueue/Transaction, SurfaceFlinger/HWC, and present, based on the events actually observed in the current trace.

The new backend service builds an observedFlow:

• lanes represent the roles observable in the current trace, such as app, render_thread, producer, buffer_queue, surfaceflinger, hwc_present.
• events represent real slice/thread/process/layer events with ts, dur, durMs, evidence source, and optional frame/layer association.
• dependencies represent confirmable dependencies, such as wakeups or critical path.
• completeness flags which signals are missing so static knowledge is not disguised as facts about this trace.

This is especially important for hybrid rendering. Producers such as WebView, Flutter, React Native, TextureView, SurfaceView, OpenGL ES, Vulkan, Camera, Video, and game engines cannot be flattened into a single “main pipeline.” SmartPerfetto now presents host HWUI and the producer separately, then explains whether there is a dependency between them based on evidence.

The value of the teaching feature is not memorizing the Android rendering pipeline for the user. It is mapping the knowledge back to the actual events in this trace. After reading, the user should know which paths have evidence in this trace and which paths cannot be confirmed because of capture gaps.

Usability Fixes in the Past Two Weeks

Beyond new features, the recent fixes mostly focus on “letting real users actually run the tool.”

Many of these issues come from real GitHub issues and local reproduction. SmartPerfetto’s difficulty is not only “can the model analyze,” but also whether the state across model, Perfetto UI, backend SSE, trace_processor_shell, browser windows, and provider protocols stays consistent. Any state drift eventually shows up as “the AI result is unreliable.”

How to Choose a Runtime Today

For typical users, Docker or the no-install package is the first choice.

Local source requires Node.js 24 LTS. The repository ships .nvmrc and .node-version, npm has engine-strict=true, and ./start.sh, ./scripts/start-dev.sh, and ./scripts/restart-backend.sh prefer switching to Node 24 through nvm/fnm.

The macOS no-install package currently uses ad-hoc signing to avoid being flagged as damaged by the system directly. Proper notarization still requires a Developer ID and a notary profile. This status should be spelled out in release notes so users do not treat Gatekeeper behavior as a backend problem.

Sponsorship and Commercial Support

SmartPerfetto’s open-source maintenance cost mainly comes from four areas: real-trace reproduction, model tokens, provider compatibility validation, and cross-platform release-package validation. The project accepts individual sponsorship, corporate sponsorship, AI Credits / token vendor sponsorship, and also commercial licensing and enterprise support consultations built around AGPL.

Individual sponsorship is suited to supporting open-source maintenance, test traces, documentation examples, and the model consumption for public demos. Individual sponsors can enter the README thank-you list; if a sponsor prefers anonymity, the project will simply record it as anonymous support.

Regular corporate sponsorship is suited to supporting long-term maintenance, issue reproduction, public documentation, and regression testing. Corporate sponsors can also enter the README thank-you list and have their company name and link displayed on the sponsor page. Deeper enterprise collaboration can be handled through commercial support, for example Android performance trace expert diagnostics, Perfetto / SmartPerfetto team training, enterprise on-prem deployment, custom YAML Skill / Strategy work, and commercial licensing or integration consulting.

LLM / API / token vendor sponsorship fits better with credits, token plans, trial accounts, or developer quotas that support SmartPerfetto. A tool like SmartPerfetto needs more than chat: it needs stable streaming output, tool/function calling, long-context recovery, structured arguments, and real trace analysis. Vendor sponsorship can be exchanged for three more direct forms of project support:

The “recommended models” piece must be grounded in real validation. Sponsorship does not change SmartPerfetto’s default provider selection, compatibility conclusions, or technical judgments. If a model is unstable in real trace analysis, tool calling, or long-running reasoning, the documentation will say so honestly. Conversely, if a vendor provides sustainable credits and the model runs stably across real SmartPerfetto analysis tasks, it earns a more prominent README position, a place in the provider configuration guide, and recommendation in the model notes.

Contact still goes through the current sponsor page and the WeChat ID 553000664 in the README. Security issues, commercial licensing, and enterprise on-prem deployment requests should be discussed separately, not mixed into public bug issues.

How to Write a Good Bug Report

SmartPerfetto bug reports live in GitHub Issues. Security issues should not be filed as public issues; use GitHub private advisory or a security email instead.

A normal bug report should at minimum include the six fields the current template asks for: Description, Component, Steps to Reproduce, Trace Information, Logs / Screenshots, and Environment. For SmartPerfetto specifically, you should also fill in version, platform, and LLM configuration. Many issues are jointly affected by the provider protocol, the model’s tool calling, trace capture quality, and runtime session state, and cannot be handled as pure frontend or backend bugs.

A locatable report typically contains these fields:

For provider-related issues, separate “the model can chat” from “the model can do stable tool calls.” SmartPerfetto depends on streaming output, tool/function calling, context recovery, and structured arguments. A provider being fine in plain chat does not mean it can finish a full trace analysis.

For trace-related issues, document the capture conditions. Android 12+ traces that include FrameTimeline are a better fit for SmartPerfetto. Scrolling, startup, ANR, and GPU/rendering each need different categories. When the corresponding data source is missing, the correct behavior is to flag the missing signal, not to fabricate a definite conclusion.

When an analysis conclusion does not match expectations, prefer pasting the evidence location from the report. For example, if a conclusion claims “main thread Runnable wait 120ms,” paste the matching Q1/C1, evidenceRefId, table row, column name, and value. That kind of feedback can turn directly into a SQL fix, a Skill branch, or a report contract fix.

What Kind of Feedback Helps Most Right Now

What SmartPerfetto is short on right now is samples that can reproduce a problem, regress against a fix, and improve a Skill. Vague evaluations are hard to convert into code fixes or Skill improvements.

More useful feedback usually looks like this:

• “In a particular scrolling trace, SmartPerfetto blames jank on the RenderThread, but a manual review says it is a main-thread Binder wait.” Include the time window, frame ID, related SQL, or screenshots.
• “In a particular report, the sentence-level data citation points to the wrong row or is missing a critical value.” Include Q1/C1, evidenceRefId, the Result ID, the HTML report, and a screenshot of the matching table.
• “A particular provider produces malformed tool-call arguments under openai-agents-sdk.” Include the runtime, provider, model ID, protocol type, and a session log.
• “A particular device/vendor trace is missing common tables, causing a Skill to fail.” Include the Android version, the trace category, the failing Skill name, and the error log.
• “A particular Result ID cannot be found in multi-trace comparison.” Include the workspace, the visibility setting, the Result ID, the current window state, and the steps to reproduce.

A good bug report should turn into one of three things: a code fix, a Skill branch, or a regression test. A screenshot plus “the result is wrong” makes it hard to tell whether the issue lives in the trace, the model, the provider, the frontend state, or a backend tool.

Closing

The direction of SmartPerfetto’s changes in the past two weeks is clear: from “add an AI conversation window inside Perfetto” toward “turn trace analysis into a queryable, reusable, shareable, regression-friendly engineering system.”

The project is still in fast development. Public APIs, internal contracts, provider presets, and the enterprise deployment surface will continue to adjust. For typical users, the most stable path is running one of your own traces with Docker or the no-install package. For contributors, the most useful input is reproducible traces, evidence that explains the failure attribution, and SQL plus regression samples that can fold back into Skills or strategies.

If you already use Perfetto in Android performance work, SmartPerfetto is worth trying again now. The introduction from two weeks ago is no longer sufficient to describe the current repository state.

The project comes from a very concrete daily workflow: you have a trace in hand, Perfetto has already exposed the facts, but moving from facts to judgment still means jumping through tables, writing SQL, matching threads, checking FrameTimeline, finding the Binder peer, and then returning to the timeline to confirm everything again. SmartPerfetto tries to turn those repeated actions into tools, so performance engineers can spend more time on judgment.

It is still in development. I am releasing it now because trace analysis grows from real samples: real devices, real vendor differences, real product traces, and real PRs all change how Skills and strategies should be written. Waiting until every capability is stable before publishing it one way would miss the stage where samples matter most.

If you often open Perfetto to inspect scrolling jank, startup, ANR, Binder, CPU scheduling, or rendering pipelines, SmartPerfetto provides a Perfetto UI with an AI Assistant. After loading a trace, you ask questions in natural language. The backend queries trace_processor_shell, invokes YAML Skills, organizes evidence, and streams conclusions plus data tables back into the browser.

Project links:

• Main repository: https://github.com/Gracker/SmartPerfetto
• Perfetto UI fork: https://github.com/Gracker/perfetto

For normal trial use, you only need the main repository. Gracker/perfetto is the frontend fork used by the perfetto/ submodule. It mainly matters to developers who want to modify the AI Assistant plugin UI.

The previous two technical articles are better for readers who want the engineering details:

• From Trace to Insight: Harness Engineering in SmartPerfetto AI Agent: the architecture of the SmartPerfetto AI Agent and the Harness Engineering process.
• SmartPerfetto Architecture Q&A: 8 In-Depth Technical Questions: Q&A around the Agent, Workflow, and YAML Skills.

Those two articles go deep into the internal architecture. Once the source is public, readers usually care more about what is actually in the repository, whether it can run, and which parts are not stable yet. This article focuses on the open-source release itself: what is open, what works today, how the internal pieces are divided, how to run it locally, and where collaboration is most useful.

What Is Open Source

SmartPerfetto open sources the full engineering stack from UI to analysis rules. It keeps Perfetto’s timeline, SQL, and trace visualization capabilities, adds an AI Assistant panel inside the UI, and uses a TypeScript backend for Agent orchestration, MCP tool calls, Skill execution, report generation, and SSE streaming output.

The repository contains these parts:

• Perfetto UI plugin and prebuilt frontend: the perfetto/ submodule integrates com.smartperfetto.AIAssistant, and the repository also includes the prebuilt frontend/ artifacts. Normal trial use does not require rebuilding Perfetto UI.
• Backend Agent runtime: scene routing, Prompt building, MCP Server, Verifier, Artifact Store, and SSE Bridge under backend/src/agentv3/.
• YAML Skill system: there are currently 165 *.skill.yaml files under backend/skills/, covering scrolling, startup, ANR, memory, CPU, GPU, Binder, rendering pipelines, and more.
• Scene strategies: 12 .strategy.md files under backend/strategies/, used to inject different analysis steps and output constraints for different questions.
• Engineering scripts and documentation: Docker Hub Compose, source-build Compose, start.sh, frontend plugin development scripts, MCP tool references, Skill guides, technical architecture docs, and rendering pipeline references.

This release opens the whole project. If a capability is not good enough yet, that is the real state of the development-stage system. There is no separate private core implementation outside the repository. The UI, backend runtime, and Skill system are already usable, while public APIs and internal protocols will continue to evolve.

Why Open Source Now

Perfetto trace analysis depends heavily on samples. Different Android versions, vendors, rendering frameworks, and business scenarios can make the same kind of jank appear in different data shapes. One person or one team cannot cover all of those branches.

During SmartPerfetto development, the same pattern appeared repeatedly: SQL can find facts, but facts do not automatically become conclusions. Many analysis mistakes look like the model is “not smart enough,” but the real cause is often that a Skill missed a branch, a table was not covered, there were too few trace samples, or a strategy assumed a specific device behavior.

Open sourcing now is about putting three things into the open and letting them be tested by real traces:

• Runnable tools: readers can load their own traces locally and verify how the AI Assistant handles real data.
• Reviewable rules: YAML Skills, scene strategies, and MCP tools are all in the repository, so the SQL and strategy behind a conclusion can be inspected, modified, and reused.
• Accumulated samples: complex performance problems need trace samples and regression tests that accumulate over time, and an open-source repository is a better place to preserve those cases.

For Android performance engineers, a trace that reproduces a problem, a more accurate Perfetto SQL query, a more reliable Skill branch, or a review that points out a wrong attribution is often more useful than a large block of feature code.

That is also the meaning of open sourcing during development: SmartPerfetto does not need to wait until every analysis path is mature before accepting outside feedback. The earlier it meets real devices, real vendor differences, and real product traces, the easier it is to add the right Skills, strategies, and regression samples.

What Problems It Targets

Perfetto’s data is powerful, but analyzing a trace is not easy. A trace that is tens or hundreds of MB can contain millions of events. Threads, slices, counters, FrameTimeline, Binder, CPU frequency, and SurfaceFlinger composition state are scattered across different tables and tracks.

The most exhausting part is usually the middle of the investigation: after seeing a frame exceed the budget, you still need to decide whether the cause is main-thread queuing, a busy RenderThread, a slow GPU, SurfaceFlinger composition blocking, or an earlier Binder, scheduling, or IO delay.

SmartPerfetto gives repeated data collection and initial judgment to the tool system, while leaving attribution that needs experience to the Agent and the engineer together.

It is suited for problems such as:

• Scrolling jank: identify janky frames, VSync budget overruns, App/SF/GPU responsibility, and main-thread or RenderThread state.
• Startup performance: split cold start, warm start, lifecycle phases, main-thread time, Binder, class loading, GC, and IO.
• ANR: locate main-thread blocking, Binder peer, lock waits, scheduling delay, and system load.
• Interaction latency: break down time from input event to frame production and then to display.
• Memory and GC: observe GC pause, memory pressure, LMK, dmabuf, and system memory state.
• Rendering pipelines: identify Standard View, Compose, Flutter, WebView, SurfaceView, TextureView, game, Vulkan, and related paths.

It does not replace performance engineers. Trace capture quality, business context, device differences, and reproduction conditions still require human judgment. SmartPerfetto is closer to an analysis assistant that can reliably run SQL, organize tables, and check missing items according to strategies.

Runtime Screenshots

At runtime, the AI Assistant can appear as a floating window inside Perfetto UI, and it can also work together with the timeline, scene markers, and analysis reports. After users load a trace, they can ask questions, inspect tool calls, expand evidence, and review conclusions in the same interface.

The floating window is useful when you want to ask follow-up questions without leaving the Perfetto timeline. It keeps the current conversation, scene entries, session history, and input box together, so you can watch the trace while deciding the next analysis direction.

The conversation timeline records Agent initialization, plan submission, Skill calls, SQL queries, and data-detail fetches. You can review the intermediate actions, not just the final conclusion.

The analysis report puts startup events, data quality, phase durations, suspicious slices, and recommendations into one result, making it easier to move from a natural-language conclusion back to concrete evidence.

Root-cause analysis tries to tie each judgment back to threads, timestamps, SQL results, and trace slices. Engineers still need to review it with business context, but they do not have to start from a blank SQL page.

Overall Architecture

The main SmartPerfetto architecture has three layers: the frontend is still Perfetto UI, the backend orchestrates the Agent and tools, and data queries are handled by Perfetto’s official trace_processor_shell.

Several boundaries in this architecture are important:

• The frontend does not analyze traces directly. It loads traces, displays the timeline, sends questions, receives SSE events, and renders results.
• The backend agentv3 is the main runtime. It handles scene recognition, Prompt assembly, MCP tool registration, Verifier checks, and report generation.
• trace_processor_shell remains the data query engine. All performance numbers come from SQL or deterministic calculations inside Skills.
• YAML Skills package common analysis flows into reusable units, so the Agent does not need to write complex SQL from scratch every time.

The design judgment is straightforward: large models are good at understanding questions, choosing directions, and organizing causal relationships. Databases and Skills are good at querying, aggregating, paginating, formatting, and repeating execution.

Skill: Turning Performance Experience into Executable Analysis Units

SmartPerfetto does not put all analysis logic into TypeScript, and it does not let the large model freely compose SQL. A large amount of domain logic lives in YAML Skills.

A Skill can declare input parameters, required tables, SQL, display columns, layered output, conditional branches, iterators, and parallel steps. The Agent only needs to call:

1

invoke_skill("scrolling_analysis", { process_name: "com.example.app" })

The backend Skill Engine executes the corresponding SQL and composite steps, then wraps the result into a DataEnvelope that the frontend can render. This has three benefits:

• Repeatable results: with the same trace and the same parameters, the Skill’s calculation path is stable.
• Renderable output: column names, data types, timestamp jumps, and duration formatting are declared in the Skill.
• Lower contribution cost: performance engineers who know Perfetto SQL can modify YAML and SQL without first understanding the whole backend codebase.

Current Skill distribution:

This Skill system is part of the SmartPerfetto runtime, and it should be separated from development-tool Skills. Claude Code Skills help developers build SmartPerfetto. SmartPerfetto YAML Skills help users analyze traces. Their names are similar, but their runtime location, inputs, outputs, and goals are different.

MCP: Letting the Agent Touch Traces Only Through Tools

SmartPerfetto exposes tools to the Agent through MCP (Model Context Protocol). Full mode exposes up to 20 tools, while Fast mode keeps only 3 lightweight tools.

Common tools fall into several categories:

• Data access: execute_sql, invoke_skill, list_skills, lookup_sql_schema, query_perfetto_source, list_stdlib_modules, lookup_knowledge, and detect_architecture.
• Planning and hypotheses: submit_plan, update_plan_phase, revise_plan, submit_hypothesis, resolve_hypothesis, and flag_uncertainty.
• Result management: fetch_artifact, write_analysis_note, and recall_patterns.
• Comparative analysis: execute_sql_on, compare_skill, and get_comparison_context.

execute_sql and invoke_skill are plan-gated. The Agent must submit an analysis plan before it can query data. This gate is not meant to freeze the workflow. It requires the Agent to state the phases, goals, and expected tools before acting, and the Verifier later checks whether key actions are missing.

Tool results are also compressed and layered. Large results go into the Artifact Store. The Agent first receives a summary, then fetches rows by page when needed. This avoids dumping hundreds of thousands of tokens from one Skill output into the context.

From Question to Conclusion

When the user enters “analyze scrolling jank,” SmartPerfetto does not send the trace directly to the large model. The trace file does not enter the prompt. The Agent only sees structured results returned by tools.

A scrolling jank analysis usually goes through these steps:

1. The frontend sends the question, traceId, and analysis mode to the backend.
2. The backend classifies the scene, such as scrolling, startup, anr, memory, pipeline, or general.
3. Prompt Builder injects the corresponding .strategy.md, architecture template, output format, and required knowledge.
4. The Agent submits a plan first, then invokes MCP tools.
5. The preferred path is invoke_skill for predefined analysis. If a Skill does not cover the case, the Agent uses lookup_sql_schema and execute_sql to add queries.
6. Skill and SQL results return as DataEnvelopes, Artifacts, tables, or summaries.
7. Verifier checks evidence support, hypothesis state, scene coverage, and common missing items.
8. The backend pushes progress, tool results, answer tokens, conclusions, and the final report through SSE.

The final user experience includes a natural-language conclusion, tables, timestamp jumps, a report URL, and a session that can continue with follow-up questions.

How to Run It

If you want to try it with your own trace, start with the shortest path. The repository already includes a prebuilt Perfetto UI, so normal trial use does not require initializing the perfetto/ submodule or preparing a local C++ build environment.

1
2

git clone https://github.com/Gracker/SmartPerfetto.git
cd SmartPerfetto

The default development environment is currently organized around macOS. Windows users should use Docker Desktop with the WSL2 backend. If you want to run source scripts, WSL2 is also recommended instead of a native Windows shell. Ubuntu and other Linux distributions still need more testing. If you hit issues, include environment details in your Issue or PR.

If you only want to run the tool first, the recommended path is the Docker Hub image. This only requires Docker Desktop/Engine and an LLM API key. You do not need Node.js. The image already includes the backend, prebuilt frontend, and a pinned trace_processor_shell:

1
2
3
4
5
6

cp backend/.env.example .env
# Edit .env and set ANTHROPIC_API_KEY;
# or configure ANTHROPIC_BASE_URL + ANTHROPIC_API_KEY for an API proxy.

docker compose -f docker-compose.hub.yml pull
docker compose -f docker-compose.hub.yml up -d

After startup, open http://localhost:10000, load a .pftrace or .perfetto-trace file, and then open the AI Assistant panel. The backend health check is http://localhost:3000/health. Uploaded files and logs are stored in Docker volumes, so they remain after container restart.

To stop the containers:

1

docker compose -f docker-compose.hub.yml down

You only need the source-build path when testing Dockerfile changes or building local code that has not been published to the image:

1
2

cp backend/.env.example backend/.env
docker compose up --build

The source-build path still uses the prebuilt frontend/ artifacts in the repository. It does not rebuild the perfetto/ submodule.

For local source checkout startup, use ./start.sh. This script uses the prebuilt frontend/ artifacts in the repository and downloads a pinned trace_processor_shell when needed:

1
2
3
4
5

cp backend/.env.example backend/.env
# Edit backend/.env and set ANTHROPIC_API_KEY;
# or configure ANTHROPIC_BASE_URL + ANTHROPIC_API_KEY for an API proxy.

./start.sh

You only need to initialize the Perfetto submodule and use the frontend development script when modifying the AI Assistant plugin UI, such as ai_panel.ts or styles.scss:

1
2

git submodule update --init --recursive
./scripts/start-dev.sh

Both local scripts start two services:

• Frontend: http://localhost:10000
• Backend: http://localhost:3000

For daily backend, Skill, strategy, or normal documentation changes, prefer ./start.sh. Switch to ./scripts/start-dev.sh only when modifying the Perfetto plugin UI.

Model integration currently focuses on Anthropic direct access or Anthropic Messages-compatible proxies. Third-party models need an adapter layer and stable support for streaming and tool/function calling. If a model can only chat and cannot reliably call tools, SmartPerfetto’s SQL queries and Skill calls cannot complete properly.

Where to Contribute

If this project becomes reliable later, the most valuable inputs will be reproducible traces, runnable SQL, Skills that explain failure causes, and regression samples that keep accumulating.

Good starting points:

• Provide small traces and reproducible problems: a trace that demonstrates the problem is easier to turn into a test and a Skill than an abstract description.
• Improve YAML Skills: add SQL, display columns, empty-data hints, vendor differences, and edge cases.
• Improve scene strategies: make scrolling, startup, anr, and other strategies cover more common analysis steps.
• Fix the Perfetto plugin UI: the AI panel, tables, charts, jumps, and scene navigation all live in the frontend plugin.
• Add regression tests: the project already has 6 canonical trace regression gates, and new failure samples can become long-term protection.

The project changes quickly during development. Before contributing, sync the main SmartPerfetto repository. If you modify the AI Assistant plugin UI, also sync the perfetto/ submodule. When you find a bug, an improvement opportunity, or have already fixed something in your fork, prefer opening an Issue or PR with the reproduction trace, expected output, and test results.

If you want to participate in Skill, Agent, or frontend development over the long term, you can use the contact information in the README to discuss collaboration. Include your GitHub account and email so it is easier to add you to core development collaboration later.

Before submitting a PR, run the checks that match your change type:

1
2
3
4
5
6

cd backend
npm run build
npm run test:scene-trace-regression
npm run validate:skills
npm run validate:strategies
npm run test:core

Different PRs need different checks. Code changes should at least run scene trace regression. Skill changes should run validate:skills plus regression. Strategy and template changes should run validate:strategies plus regression. Type or build issues should add npx tsc --noEmit.

License and Current Boundaries

SmartPerfetto core code uses AGPL-3.0-or-later. The perfetto/ submodule is a fork of Google Perfetto and continues to use Apache-2.0. Before internal enterprise modification, external service deployment, or commercial integration, you should carefully understand the obligations of AGPL. If you need commercial licensing without AGPL obligations, use the contact information in the README.

Open sourcing the full project does not mean every analysis path is complete. Several runtime boundaries remain:

• It is best suited for Android 12+ traces that include FrameTimeline data.
• The model needs stable support for streaming and tool/function calling.
• Third-party model compatibility still needs more real testing. For now, Anthropic direct access or a stable Anthropic-compatible proxy is recommended first.
• When trace capture misses required fields, many conclusions can only degrade into missing-data explanations.
• Agent output still needs engineer review, especially for vendor customization, non-standard rendering paths, and complex cross-process waits.

After open sourcing, SmartPerfetto puts repeatable trace queries, scene steps, data presentation, and quality checks into the repository. Performance analysis will not become a single prompt. Complex problems still need Perfetto, SQL, trace evidence, and engineering experience.

SmartPerfetto aims to turn that experience into engineering artifacts that can be modified, reviewed, reused, and protected by regression tests.

Discussion Group

If you are trying SmartPerfetto, want to report bugs, discuss trace analysis results, submit PRs, or help build Skill, Agent, and frontend capabilities, scan the code to join the discussion group:

If the group QR code is full or expired, add me on WeChat at 553000664 and include SmartPerfetto in the note. I will invite you to the group.

Starting from Ming-Chi Kuo’s April 27, 2026 supply-chain report about an OpenAI phone, this article looks at the possible system shape after phones and AI merge, from the perspective of someone who works on Android phones.

Introduction: This Is Not Just a New Phone Problem

The most interesting part of Kuo’s report is not the 2028 mass-production window. It is not whether MediaTek, Qualcomm, or Luxshare ends up in which supplier position either.

It pushes the question into the system layer:

If the user’s primary goal shifts from opening apps to completing tasks, what should a phone operating system look like?

That sounds like a UI question. From an Android practitioner’s perspective, it touches the whole system structure: Launcher, notifications, permissions, IPC, app capability declarations, model runtime, TEE, device-cloud sync, task state machines, audit logs, payment confirmation, and developer revenue sharing all need to be reconsidered.

Over the past few years, most discussions about AI phones have stayed at the feature layer. Vendors talk about AI photo editing, AI erase, AI summaries, AI search, AI assistants, and AI briefings. All of these can fit inside today’s Android or iOS architecture.

If OpenAI really builds an AI Agent phone, it will move the question from “how many AI features can be added to a phone” to “should the phone’s first entry point still start from app icons?”

The main line is this: Agent OS looks like a structural migration of mobile OS after the graphical interface era. The foreground moves from an app grid to a task stream. The background moves from app-owned capability to authorized capability. The system moves from managing processes and windows to managing tasks, context, and responsibility.

OpenAI does not have to build on Android, but Android is the more likely path because it inherits hardware adaptation, drivers, app compatibility, and supply-chain experience. If OpenAI wants to design the first screen, permissions, and task flow completely around Agent OS, it may also choose an “Android-compatible but not quite Android” path, or even build a Linux-based system and fill the service gap with the web, cloud execution, and an app compatibility layer.

Different OS choices will change go-to-market speed. They will not change the five things Agent OS must solve:

1. The phone must continuously understand the user’s current state.
2. Apps must move from foreground entry points to background capability providers.
3. The system must have a task runtime that is recoverable, cancellable, and auditable.
4. Device-side and cloud-side execution must be split by data sensitivity and real-time needs.
5. Every cross-app, cross-device, and cross-cloud action must have permission, responsibility, and rollback boundaries.

Without these five things, an AI phone is still just “a phone with an AI assistant.” Once they become system constraints, the phone starts moving toward Agent OS.

Table of Contents

The article follows six major questions.

1. The starting point: OpenAI’s phone does not change hardware first. We first separate several possible OS paths for an OpenAI phone, then explain why the phone is still the Agent’s state center.

2. Entry migration: from app grid to task stream. Apps will not disappear, but their entry role will be downgraded. The real change lands on OS objects such as Task, Capability, Context, Artifact, Grant, and Audit.

3. The Android path: from super app to system service. Android will not be rebuilt from scratch in the short term. Agent OS is more likely to grow out of system services, A2A, AppFunctions, GUI Agents, and a Capability Router.

4. OpenAI’s fork: Android-based or not Android-based. A non-Android route is cleaner, but it must rebuild apps, identity, payment, service networks, and responsibility boundaries. The Android route is faster, but easier to drag back into the old app model.

5. Device-cloud, hardware, industry, and timeline. Device-cloud division of labor, data classification, on-device models, SoCs, five groups of players, the supply chain, and the 2026-2032 timeline will decide whether Agent phones can move from demos into daily use.

6. Product shape, market split, and responsibility distribution. Finally we return to the first screen, the China market, developer relations, responsibility distribution, and what Android practitioners should watch.

1. The Starting Point: OpenAI’s Phone Does Not Change Hardware First

Kuo’s report points to several concrete directions: OpenAI is working with MediaTek and Qualcomm on phone processors; Luxshare handles assembly and system integration; hardware co-design may be involved; specifications and suppliers may converge by late 2026 or Q1 2027; the target is mass production in 2028.

If this is understood only as “OpenAI wants to build an iPhone competitor,” the analysis will go off track. Phone hardware is already a mature industry. OpenAI’s hard part is rarely bill of materials, cameras, industrial design, or finding a contract manufacturer. The hard question is: why would users hand their primary device to an AI company that has no app distribution power, no local-life service network, no payment network, no carrier channel, and no long-term phone support experience?

The answer cannot simply be “a stronger model.” A phone is not a benchmark device for models. Users handle identity, money, work, social relationships, transportation, health, family, and content consumption on phones every day. OpenAI’s opportunity is on the other side: bypass the fight for dwell time inside each app, and compete directly for task interpretation and task execution.

But “building a phone” can mean four different choices.

The first choice is standard Android. OpenAI can take the AOSP/GMS stack, deeply customize Launcher, SystemUI, permissions, notifications, and the default assistant, keep Android app compatibility, and make the Agent a system-level entry point. This is the fastest path, and it looks most like a phone. The risk is limited control. Android’s security model, background restrictions, app permissions, Play policies, and OEM customization boundaries will all limit how far OpenAI can rewrite the user experience in one step.

The second choice is a deep AOSP fork. It still runs Android apps, but OpenAI defines the framework, default entry point, capability registration, task management, and cloud sync. Overseas, it can negotiate Google services. In China, it would almost need to build a separate service system. This path is feasible in engineering terms, but commercially difficult. Android app compatibility touches APKs, Google Play, push, maps, payment, accounts, subscriptions, anti-cheat, and vendor certification.

The third choice is “Android compatibility layer plus self-built Agent OS.” The bottom layer may still be Linux. The foreground is a task-flow system. Android apps run inside an isolated container or compatibility environment, similar to how desktop systems have run mobile apps before. OpenAI gets more design freedom in this system. It can treat apps as capability containers and weaken their UI as the main subject. The cost is higher compatibility work, power consumption, background limitations, and resistance from app vendors.

The fourth choice is a new OS that is not based on Android at all, and the first-generation device may not even be a traditional phone. OpenAI’s earlier hardware direction with Jony Ive’s team has often been understood as a more AI-native device. That does not conflict with Kuo’s phone report. OpenAI can first build a portable device with a weaker screen, stronger conversation, and heavier cloud execution, or build a phone-shaped device that does not aim for full app compatibility. It is cleaner, but it is hard to become the user’s primary phone.

My leaning is this: if the goal is 2028 mass production and high-end replacement demand, OpenAI probably will not completely abandon the Android ecosystem. The more realistic path is an AOSP or Android-compatible branch, plus a heavily customized Agent OS foreground. But this is a probability judgment, not an assumption. The following sections first analyze the Android route, then the non-Android route.

Before route analysis, one concept should be separated: OpenAI may not want to “make an Android phone.” It is more likely trying to build a task entry point that makes Android and iOS feel old. If users accept that entry point, first-generation hardware sales should not be the only metric. It will force every phone vendor to answer the same question:

Why should users still search through dozens of icons instead of seeing tasks that are happening, waiting for confirmation, already done, or ready to resume?

Why the Phone Is Still the Agent’s State Center

The view that “the Agent can just live in the cloud” sees reasoning capability but misses the phone’s position in user life. Cloud models are larger, tools are richer, and costs can be centrally optimized, but the cloud lacks the user’s current state.

Completing tasks depends on language understanding, but also on the user’s immediate state. State is broader than a prompt plus static data such as contacts, calendar, and photos. It points to what is happening now, and how far the user is willing to let the system act on their behalf.

The phone is exactly the device with the most complete version of that state.

It knows where the user is, what they are looking at, what notification just arrived, which app was opened a minute ago, whether Bluetooth earbuds are connected, whether the watch detected activity, whether the screen is locked, whether payment credentials are available, whether the user just entered an airport, whether the car is parked nearby, how long until the next meeting, and whether family members are in the same location-sharing circle.

The cloud does not naturally get this state. It can have long-term memory, knowledge bases, browsers, code sandboxes, and powerful models, but it cannot see the scene in the user’s hand. It does not know whether the user can speak right now, which button on the current screen can be tapped, or whether biometric confirmation can be completed. If the cloud wants to behave as if it is present, it must get live state through the phone.

This is why OpenAI building a phone makes sense. However strong the ChatGPT app becomes, it is still an app: it appears when opened and leaves when closed. It has no system notification entry, no default right to confirm payments, no continuous semantic access to camera and screen, no cross-app task state, and no long-term permission to manage device state.

The boundary between an Agent phone and an AI app is here.

An AI app waits for the user to open it. An Agent phone observes events at the system layer, organizes tasks, waits for the right time, requests authorization, executes actions, and records results. The former is a very capable tool. The latter is a system-constrained agent.

Ordinary users will not describe it this way, but their behavior will push this change. When someone says, “Book my flight and hotel to Tokyo next Wednesday, keep it under 8,000, avoid red-eye flights, and send the itinerary to my colleagues,” the goal is to have a task decomposed, compared, confirmed, paid for, synced, and tracked. The goal is not to switch among Trip.com, Fliggy, maps, calendar, WeChat, email, and the company expense system.

When a parent says, “Check the class group and tell me what materials my child needs to submit this week,” the goal is to produce an executable checklist from group messages, the school mini-program, photos, calendar, and to-dos, then remind before the deadline. The goal is not to open WeChat and scan 200 messages, or ask a model to summarize a chat.

When a creator says, “Turn today’s footage into three short videos for Douyin, Xiaohongshu, and Channels,” they are not asking to open Jianying and tap buttons. They want material understanding, platform specs, copywriting, style selection, copyright checks, publishing plans, and metrics review.

None of these are single-app problems. They span apps, accounts, devices, cloud tools, and real-world fulfillment. The phone is the entry point because it holds state. The cloud is the amplifier because it carries long tasks. Apps are capability sources because services still sit behind apps.

So App -> Agent means the app’s foreground status declines while its capability status rises. The app itself still exists.

2. Entry Migration: From App Grid to Task Stream

Many Agent OS discussions speak too absolutely: Agents will replace apps. That judgment is too crude.

As described above, apps will not disappear. The reason is simple: apps carry account systems, business relationships, payment risk control, content rights, private-domain relationships, membership benefits, customer support responsibility, and regulatory obligations. An Agent can operate a food-delivery app for the user, but restaurant supply, delivery networks, after-sales rules, coupons, invoices, and food safety responsibility still sit with the platform. An Agent can help book a hotel, but inventory, price, cancellation policy, and credit guarantees still sit with OTAs and hotel groups.

The change happens at the entry layer.

The old phone is an app icon grid: users pick an app first, then find the function inside the app. That model fits the era of humans actively operating software. The foreground of Agent OS is more like a task stream: users express goals first, and the system chooses capabilities. Apps move from “destination” to “capability source.” Users may not know which app was finally called, but the system must know, explain, authorize, audit, and assign responsibility.

Kuo’s concept image compresses this into three lines:

• App -> Agent
• Icon -> Task
• Grid -> Stream

These three lines work as product direction, but engineering must separate them.

App -> Agent is not about wrapping every app in a chat UI. An Agent needs to understand the capabilities an app exposes, input constraints, output format, execution risk, and charging rules. For Android, this corresponds to a combination of AppFunctions, Intent, ContentProvider, ShareSheet, Shortcut including App Shortcuts and Sharing Shortcuts, Notification Action, Credential Manager, Wallet/Payment SDKs, web APIs, and vendor-private interfaces. Slice has been marked deprecated by Google, so it should not be included in new designs.

Icon -> Task is not about replacing app icons with task cards. Tasks need a state machine: created, planned, waiting for authorization, running, waiting for external response, failed, canceled, completed, rolled back. They also need Artifacts: tickets, orders, documents, calendar events, images, invoices, meeting notes, routes, code diffs. Users should not only see a spinning chat bubble.

Grid -> Stream is not about making the home screen an information feed. A task stream must be controllable, collapsible, mutable, pausable, and traceable. Otherwise it becomes an even noisier notification center. An Agent that can appear proactively will be turned off quickly if it has no budget or restraint.

The mobile OS change hides behind these three conversions.

In the past, the OS managed processes, windows, permissions, and notifications. Agent OS must also manage tasks, context, capabilities, memory, and responsibility. Apps can continue to exist, but the first interface users face each day may no longer be a display case of apps. It may become a workbench of tasks.

The OS Object Model Behind One Task

If we only say App -> Agent, it quickly becomes an abstract judgment. In Android engineering language, a cross-app task must be split into at least the following objects.

The first object is Intent. Here it means the user’s goal, not Android Intent. It can come from voice, text, screenshot, camera, notification, clipboard, or an external device. If the user says “help me reimburse this invoice,” the system cannot just throw the sentence to a model. It must first identify the task type, involved apps, required accounts, possible credentials, company policy, network needs, and whether the user must be present to confirm.

The second object is Context. Here it means the minimum scene information required by the task, not Activity Context. A reimbursement task may need the invoice image, the approval entry in WeCom or Feishu, company reimbursement rules, invoice title, user identity, and the last template used for a similar reimbursement. The system must trim this information to what is sufficient for the task, without handing the whole album, whole chat history, or all company documents to the cloud.

The third object is Capability. The system must know what capabilities can finish the task: OCR, invoice verification, the enterprise app’s reimbursement AppFunction, browser form filling, company intranet API, file upload, and message notification. Each capability has a source, permission, cost, reliability, execution duration, and risk level.

The fourth object is Plan. The Agent will break the task into steps, but the plan should not live only inside model context. It should be recorded by the system as an inspectable structure: read invoice, extract fields, verify authenticity, open reimbursement entry, fill form, upload attachment, wait for user confirmation, submit, save receipt. Then the user, system, and developer can know where the task is.

The fifth object is Permission Grant. Traditional app permission asks “can this app access the camera?” Agent permission asks “can this reimbursement task read these three photos, access this company reimbursement entry, submit the form, and must submission be confirmed first?” It is shorter, finer-grained, and more dependent on task state than app permission.

The sixth object is Artifact. Task outputs include reimbursement drafts, submission receipts, invoice fields, approval numbers, failure screenshots, and operation logs, not just an answer sentence. These Artifacts must enter history and be available to later tasks.

The seventh object is Audit. What the Agent did, whom it called, which data it used, which step the user confirmed, which step was automatic, and where failure happened must all be traceable. Without Audit, once the Agent makes a mistake, neither users nor vendors can determine responsibility.

This object model relates to existing Android objects, but cannot be fully reused.

Activity is for UI. Service is for background execution. Broadcast is for event distribution. ContentProvider is for structured data. Intent is for component invocation. They solve how apps cooperate with each other. They do not solve how a cross-app goal keeps running, waits for user confirmation, migrates between cloud and device, or recovers after failure.

Agent OS needs to add task objects above these traditional objects, wrapping Activity and Service in a higher-level task semantics.

One way to understand the relationship is:

1
2
3
4
5
6
7
8
9
10
11

Old Android objects:
App / Activity / Service / Intent / Permission / Notification

New Agent OS objects:
Task / Step / Capability / Context / Artifact / Grant / Audit

Migration direction:
Users no longer directly manage Activity stacks; they manage Task state.
Apps no longer expose only UI and Intent; they expose Capability.
Permissions are no longer only long-term app grants; they are temporary Task grants.
Notifications expand from app events into Task progress.

This is also why Agent OS cannot be done by an independent app. A third-party app can maintain its own Tasks, but it has no authority to unify all apps’ capabilities, permissions, context, and audit records. Only the system layer can do that.

The Relationship Between Task Stream and Notification Center

The first thing the system layer must do is organize fragmented events into tasks. This is also the part most easily confused with the notification center.

Many vendors will start from notification center changes because notifications are natural entries for “something happened.” Directions such as Samsung Now Brief and Now Bar have value for exactly this reason. They try to reorganize fragmented events into “things the user needs to handle today,” and their value comes from reorganizing events into tasks.

But a task stream cannot simply equal a notification center.

A notification is an event an app sends to the user. A task is a goal the user or system needs to complete. Notifications can trigger tasks, and tasks can generate notifications, but they are not the same object.

For example, an airline app pushes “flight delayed by two hours.” That is a notification. After Agent OS receives it, it can generate a task: “Should pickup time, hotel check-in, meeting schedule, and colleague notification be adjusted?” This task reads calendar, maps, hotel orders, chat contacts, and company meeting arrangements, then asks the user to confirm. The notification is only the starting point. The task is the object the system must manage.

Another example: a school group sends 20 messages, and only one is about materials due tomorrow. A traditional notification center can only show app pushes. An AI summary can summarize the chat. Agent OS must go further and generate a task: “Print and fill the form before 9 p.m. tonight, and take it to school tomorrow morning.” It needs to connect files, printer, calendar, reminders, and family members.

So a task stream is more like a lightweight project manager. Its objects are life and work to-dos, not a notification list.

This brings a product constraint: the task stream must be restrained. A phone that proactively creates 30 tasks a day will be disabled quickly. The system needs reminder budgets, interruption levels, quiet hours, family/work modes, and user feedback. The difficulty of a task stream is filtering out tasks that should not appear.

3. The Android Path: From Super App to System Service

Android has natural advantages for Agent OS, and obvious burdens.

The advantages are openness, modifiability, broad hardware coverage, OEM access to system permissions, and many reusable framework components. Binder fits high-frequency IPC on the same device. PackageManager fits capability registration. PermissionController fits authorization. ActivityTaskManager and WindowManager fit foreground/background state. NotificationManager fits long-task visibility. AppSearch fits local indexing. Keystore and StrongBox fit credential protection. JobScheduler fits long-task scheduling. ContentProvider fits structured data access.

The burdens are fragmentation, historical permission baggage, the split between Google services and AOSP, too many vendor-private interfaces, complex background restrictions, and top apps’ natural suspicion toward automation. An Android Agent that wants to move from “can tap the screen” to “can reliably finish tasks” must cross three technical paths.

The first path is GUI Agent. It uses screenshots, accessibility, input injection, Virtual Display, UI hierarchy, OCR/VLM, and similar methods to simulate a human operating apps. The advantage is that it covers long-tail apps without developer cooperation. The shortcomings are also obvious: it is slow, vulnerable to UI changes, and easily touches security boundaries. Payment, social, banking, and gaming platforms will naturally guard against it.

The Doubao phone pushed this path onto real devices. It showed the industry both usability and boundaries. System-level permissions, Virtual Display, on-device visual understanding, and cloud OpenClaw/ArkClaw together can complete many cross-app actions. But third-party app restrictions also show that “tapping the screen like a human” cannot support the long-term main path.

The second path is structured capability. Android AppFunctions, Apple App Intents, vendor SDKs, MCP tools, and private APIs all fall into this category. It requires apps to actively declare “what I can do,” and provide parameters, permissions, return values, and risk levels. The advantage is stability, speed, and auditability. It fits operations such as payment, sending messages, creating calendar events, generating documents, and submitting orders. The drawback is developer cooperation, and top apps will not easily give up foreground entry.

The third path is Agent protocol. A2A solves how Agents discover each other, delegate tasks, pass messages, and deliver Artifacts. MCP solves how Agents use tools and data sources. On phones, they will not replace Android IPC, but coexist at different layers. On the same device, Binder is a better fit. Across devices and the cloud, network protocols fit better.

A more reasonable short-term Android structure looks like a router:

1
2
3
4
5
6
7
8
9
10

User goal
  ↓
On-device intent recognition and risk judgment
  ↓
Capability routing
  ├─ AppFunctions / App Intents / private APIs: preferred
  ├─ A2A / MCP: cross-Agent and cross-cloud tools
  └─ GUI Agent: long-tail apps and temporary compatibility
  ↓
Task state, user confirmation, audit log, result archive

This is also the mistake Android vendors should avoid in the short term: turning Agent OS into a super app with all permissions. A super app can make demos, but it cannot support a system architecture. The scalable long-term approach is to split the Agent into system services, capability declarations, task state machines, data classification, and user-visible control.

Where Existing Android Components Can Migrate

Android does not start from zero. From a platform engineering perspective, many Agent OS parts can migrate from existing system abstractions.

PackageManager can continue discovering installed apps, but it needs to extend capability indexing. In the past, PackageManager cared about Activity, Service, Receiver, Provider, Permission, and Feature. Agent OS also needs to know which AppFunctions an app declares, whether it provides an AgentCard, which schemas the system can call, which capabilities need human confirmation, and which capabilities can only be used under enterprise policy.

PermissionController can continue handling authorization UI and permission records, but it needs task-level grants. Traditional permission is often “allow this app to access the camera.” Agent permission is more like “allow this task to read these three images and upload them to this service, but show a preview before upload.” This kind of grant should not be permanently attached to an app. It should bind to Task, Step, and Artifact.

ActivityTaskManager can continue providing foreground/background state, task stacks, and window information, but Agent OS needs usable semantic snapshots, not raw copying of the user’s current screen. The system should expose structured summaries: current app, page type, actionable elements, sensitive regions, whether the user is typing, and whether a payment, verification code, or privacy screen is visible. Exposing raw screen content to the cloud brings high risk.

NotificationManager can become the natural outlet for task state. A long task should not stay only inside an assistant app. It should enter system notifications, lock screen, watch, and car. But notifications also need to upgrade: not just title and body, but state, progress, next action, risk prompt, pause/cancel/confirm buttons.

AppSearch can become part of local semantic indexing. The device needs to quickly find notes, files, calendar events, contacts, historical tasks, common places, and user preferences. AppSearch fits local indexing, but sensitive data must be split by policy. The system cannot dump everything into an index that an Agent can read freely.

ContentProvider remains the base for structured data access, but its permission model needs to be finer. An app may allow the Agent to read one order status, but not the full order history. It may allow reading an attachment in one chat, but not the whole chat context.

Keystore, StrongBox, TEE, and secure elements become more important. On an Agent phone, credentials, payments, Passkeys, enterprise certificates, private memory, and sensitive vector indexes all need hardware-level protection. The Agent can prepare actions, but final signing and confirmation should happen inside a security boundary.

JobScheduler, WorkManager, AlarmManager, and push can support long-task scheduling, but Agent tasks are more complex than ordinary background tasks. A task may run in the cloud while the device only waits for callbacks. It may execute halfway on device, lose network, and resume later. It may migrate across devices. The system needs to connect task state with the scheduling system.

RoleManager and default assistant roles will become key entry-point battlegrounds. Android has used RoleManager to manage default dialer, SMS, browser, assistant, and similar roles; input method uses a separate InputMethodManager path. Agent OS may need “default Agent,” “default task manager,” and “default capability router.” This will create a new contest among platform, OEM, and third-party assistants.

Put these components together and Android can take a pragmatic short-term evolution path:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

SystemUI / Launcher
  Show Task Stream, confirmation cards, and task history

AgentManagerService
  Manage Agent, Task, Capability, Grant, and Audit

CapabilityRegistry
  Aggregate AppFunctions, AgentCard, MCP tools, system capabilities, private extensions

ContextService
  Provide minimized semantic snapshots, not unbounded screen reading

PolicyService
  Handle data classification, user consent, enterprise policy, compliance rules

ModelRuntimeService
  Schedule on-device LLM/VLM/embedding/rerank/intent models

CloudBridge
  Sync Task, Artifact, Push, and Audit with cloud Agent runtime

This structure does not require Android to be rebuilt immediately. It is more like adding a set of high-privilege system services to the current Android stack, then gradually opening developer interfaces. OEMs will likely implement it privately in the short term. Only later might AOSP, GMS, or industry protocols absorb it.

The Right Position for A2A on Android

A2A deserves a separate explanation because it is not just another “tool-calling” protocol.

A2A stands for Agent2Agent. It was launched by Google with 50+ partners in April 2025 and transferred to Linux Foundation governance in June 2025. Its official positioning is clear: allow Agents from different vendors, frameworks, and runtime environments to discover each other, delegate tasks, exchange progress, and deliver results.

Its design assumption is that Agents are black boxes to each other. A travel Agent does not need to know which model, prompt, tool, or database a reimbursement Agent uses internally. It only needs to know what the other side can do, how to authenticate, how to submit a task, what state the task is in, and what result will be returned.

This differs from MCP. MCP solves how an Agent calls tools, resources, and data sources. For example, an Agent uses MCP to access the filesystem, database, browser, Jira, or Feishu docs. A2A solves how an Agent calls another Agent. The former is more like a tool socket. The latter is more like a task delegation language.

This distinction is useful on phones.

If the user says, “Plan my Tokyo trip next month, keep it under 8,000, avoid red-eye flights, and send the itinerary to my colleagues,” the phone-side system Agent can read calendar, location, payment-confirmation state, and contacts. But it does not have to do everything itself. It can delegate flight and hotel comparison to a travel Agent, visa material checking to a document Agent, company travel rules to an enterprise Agent, and leave final confirmation and payment on the phone.

If every Agent has a private interface, this quickly becomes unmanageable. A2A tries to provide a common vocabulary.

The first term is AgentCard. It is the Agent’s capability card: who I am, what I can do, which inputs and outputs I support, what authentication I need, whether I support streaming, and whether I support push after task completion. On Android, it resembles a Manifest for Agents. An app or system service can declare “I have a travel Agent,” “I have an image-editing Agent,” or “I have an enterprise reimbursement Agent.” The system indexes these cards.

The second term is Task. A2A Task is a stateful work unit, not a single API request. It can be submitted, working, input-required, auth-required, completed, canceled, failed, rejected, unknown, and so on. This fits phones because real tasks often span time: ticket booking waits for price, food delivery waits for rider, approval waits for manager, video export waits for processing, reimbursement waits for submission. Users need to see task state, not just a model reply.

The third set of terms is Message / Part / Artifact. Message is a conversation turn in the task process. Part can be text, file, or structured data. Artifact is the deliverable. Phone Agents need to pass images, PDFs, spreadsheets, orders, calendar events, location, credential references, and operation results. Artifact is especially suitable for Agent OS because task results must be saved, shared, audited, and referenced later. A travel task’s Artifacts can include itinerary, hotel candidate table, budget table, calendar events, and payment confirmation record.

The fourth capability is Streaming and Push. Many phone tasks do not finish in seconds. Users cannot stare at a chat window while a cloud Agent completes research, comparison, approval, or video processing. A2A’s streaming updates and asynchronous callback semantics fit the pushback of “task started,” “more information needed,” “authorization needed,” “completed,” or “failed reason” to phone notifications, lock screen, and task stream.

So A2A’s value for Agent OS is not that it adds one more network protocol. Its value is standardizing several of the hardest objects in Agent collaboration: who can do what, what state the task is in, how the process communicates, and how results are delivered.

But A2A should not be inflated into the whole answer for Android Agent OS.

First, A2A is natively aimed at cross-system, cross-network, cross-vendor Agent communication. It fits a phone-side Agent calling cloud OpenClaw, ChatGPT Agent, enterprise Agent, car Agent, or home Agent, and it fits Agent delegation between companies.

Second, on the same Android device, A2A semantics are worth borrowing, but the transport layer should not simply copy HTTP/JSON-RPC. Android’s local IPC path is Binder. Making local Agents go through loopback HTTP every time adds latency and bypasses existing UID, SELinux, permission, lifecycle, and other security models. A better direction is A2A over Binder: keep AgentCard, Task, Message, Artifact semantics, but use Binder for transport, with permissions and audit handled by system services.

Third, A2A needs a division of labor with AppFunctions. AppFunctions are structured functions for a system Agent to call local apps, such as createNote(title, content), searchPhotos(query), and sendMessage(contact, text). A2A is for one Agent delegating a task to another Agent, such as “organize this batch of photos into a travel album” or “handle reimbursement according to company rules,” where autonomy, follow-up questions, and long execution are needed.

A more reasonable Android stack looks like this:

1
2
3
4
5
6
7
8
9
10
11

User goal
  ↓
Phone-side system Agent
  ├─ AppFunctions → local app atomic capabilities
  ├─ MCP / private tools → cloud or local tools
  ├─ GUI Agent → long-tail app compatibility path
  └─ A2A → local / cloud / cross-device Agent collaboration
        ├─ Cloud long-task Agent
        ├─ Enterprise Agent
        ├─ Car / home Agent
        └─ Third-party service Agent

Fourth, A2A still lacks pieces a phone OS must supply. It can describe Task state, but it does not own Android permission UI. It supports async callbacks, but does not guarantee strict event replay across all implementations. It can pass Artifacts, but it does not define data classification, device-cloud boundary, payment confirmation, or failure rollback for the system. Agent OS must add Policy, Consent, Audit, Retry, Idempotency, and user-visible control above A2A.

Therefore the better position for A2A on Android is this: it does not replace Binder, AppFunctions, or the permission system. It should become the public semantic layer for Agent collaboration. Android’s job is to translate that semantics into its own system services, IPC, task stream, and permission model.

The Three Technical Routes Will Coexist for a Long Time

After separating GUI Agent, AppFunctions, and A2A, the conclusion is clear: all three will coexist for a long time, but they occupy different positions.

GUI Agent is the compatibility layer. It solves the problem of “developers have not cooperated, the app exposes no capability, and the user wants the task done now.” It fits long-tail apps, transition periods, and low-risk operations visible to the user. It should not be the default path for high-risk actions such as payment, messaging, settings changes, data deletion, or ordering.

AppFunctions / App Intents are the local capability layer. They let apps tell the system in a structured way: I can create notes, send messages, create events, search products, submit orders, generate images, or export videos. They need developer cooperation, but once scaled, they are more reliable, faster, and safer than GUI Agent.

A2A / MCP are the collaboration and tool layer. MCP targets tools and data sources. A2A targets discovery, delegation, and delivery among Agents. A phone-side Agent needs similar protocols to coordinate with cloud Agents, car Agents, computer Agents, and enterprise Agents. Protocols will not replace local APIs, but they make cross-device and cross-cloud tasks describable.

These three paths need a Capability Router. The user only expresses a goal. The system decides which path to use based on task type, risk level, available capabilities, network state, privacy level, and cost.

For example, “make this photo a little brighter” can be handled by an on-device model.

“Turn today’s meeting recording into minutes and send it to a Feishu group” should prefer transcription, documents, Feishu API or AppFunction, then ask the user to confirm sending.

“Compare three family trip routes for summer vacation, keep it under 20,000, and consider that the elderly cannot walk too much” can run long cloud queries and comparisons, while the phone syncs family calendar, location preferences, payment confirmation, and final choice.

“Cancel a subscription that is about to renew” is best if the app exposes structured capability. If not, GUI Agent can operate the settings page or webpage. When payment and account changes are involved, user confirmation is required.

This is why Agent OS is harder than an assistant app. An assistant app only needs to be a model entry point and apologize after failure. Once Agent OS promises execution, it faces real-world side effects. It cannot only be smart; it must be controllable, explainable, and recoverable.

Short-term Android Agent OS Architecture: System Services Are the Main Battlefield

After the three Android paths, we can converge them into one system architecture.

This structure should not be treated as the final Agent OS architecture. It is more like a pragmatic Android evolution path from 2026 to 2028: keep Android and the kernel mainline, and add a group of Task-centered system services across SystemUI, Framework, permissions, notifications, on-device model runtime, and cloud Agent Runtime.

The architecture is easy to read.

The first layer is Agent UI / Task Stream. Users see task state: running, waiting for confirmation, completed, failed, recoverable. They do not see model calls. Home screen, lock screen, notifications, floating entry points, and multi-device handoff become different outlets for task state.

The second layer is Agent Runtime. This is the semantic layer Android lacks most in the short term. TaskRuntime manages the state machine, Artifacts, cancellation, recovery, and rollback. ContextService provides minimized scene snapshots. CapabilityRouter routes among AppFunctions, A2A, MCP, and GUI Agent. MemoryService manages short context, long-term preferences, and deletable memory. Policy / Consent / Audit handles authorization, consent, risk classification, and responsibility records.

The third layer is Android Framework. Android does not need to write an OS from scratch. PackageManager, PermissionController, ActivityTaskManager, NotificationManager, AppSearch, ContentProvider, Credential, Keystore, StrongBox, JobScheduler, and RoleManager can all continue to work. The change is that these old objects are wrapped by a higher-level Task semantics.

The fourth layer is Sandbox & Runtime. ART still runs Java/Kotlin apps. Native runtime still handles high-performance execution. Isolated process, Privacy Sandbox SDK Runtime, Microdroid, or similar virtualization capabilities can carry stronger isolation. On-device model runtime, embedding, VLM preprocessing, and GUI automation sandbox should also be scheduled at this layer.

The fifth layer is HAL / Kernel / Hardware. It provides NPU/GPU/CPU, memory tiers, low-power sensor hub, TEE/SE, Binder, SELinux, cgroup, seccomp, mmap CoW, TEE drivers, and other resource and security foundations. The kernel should not carry Agent task semantics. It should be responsible for resources, security, isolation, and performance.

On the right is Cloud Agent Runtime. An Agent phone must be a device-cloud organism. The cloud handles long-task sandboxing, browser, code tools, document tools, long-term memory, A2A gateway, push callbacks, and data policy. The device handles live context, privacy, confirmation, and braking. What synchronizes between them is Task, Artifact, authorization, audit, and state, not an unbounded chat transcript.

This structure contains several key choices.

First, put Task Runtime in a first-class position. The foreground of Agent OS is the task stream; the chat box is only one entry point. The system must manage task state, Artifacts, progress, failure, cancellation, rollback, and history. Without Task Runtime, Agent OS easily falls back into “assistant app plus notification.”

Second, separate Policy / Consent / Audit from Access manager. Agent operation on a phone is not ordinary permission asking. It needs to answer: is this task allowed, does this step require confirmation, is the result traceable, can the user revoke it, and who is responsible? Payment, messaging, file deletion, order submission, and external publishing all need audit and consent. The permission system can answer “can it access,” but cannot fully answer “should it execute.”

Third, distinguish Android, GMS, and vendor-built capabilities. AICore is Google’s system-app-based on-device foundation model service for Pixel and partner OEMs, hosting Gemini Nano. It is not part of AOSP, and Chinese ROMs do not naturally have it. Also note that Apple Private Cloud Compute (PCC) and Android Private Compute Core (also abbreviated PCC, hosted by Android System Intelligence) are different systems. They should not be conflated. Chinese vendors are more likely to replace them with self-built model services and local compliance clouds.

Fourth, do not describe ART as Agent Runtime. ART will continue to handle app execution, JIT/AOT, profile optimization, and part of the process model. Agent Runtime is unlikely to evolve directly from ART. A more likely structure is Agent Runtime as system services and sandbox runtime, built above ART, native runtime, isolatedProcess, SDK Runtime, and Microdroid/virtualization capabilities.

Fifth, split Memory. Agent “memory” has at least four types: current context snapshot, long-term user preferences, intermediate task state, and model KV/cache plus embedding indexes. Their security level, lifecycle, storage location, and deletion mechanism are different. Calling all of them Semantic memory hides engineering boundaries.

The basic judgment for short-term Android Agent OS is that the main battlefield is not the kernel, not Launcher alone, and not a single super-assistant app. It is between Framework/SystemUI/privileged services and cloud Agent Runtime.

At the Engineering-Service Level, at Least Four Groups Are Needed

If this architecture is further decomposed into Android engineering services, the Framework layer needs at least four groups.

The first group is Task services:

1
2
3
4
5

TaskManagerService
TaskStateStore
ArtifactProvider
TaskHistoryProvider
TaskNotificationAdapter

TaskManagerService creates, resumes, cancels, and schedules tasks. TaskStateStore saves task state and steps. ArtifactProvider manages task outputs. TaskHistoryProvider lets users and the system query history. TaskNotificationAdapter syncs tasks to notifications, lock screen, watch, and car.

This group decides whether Agent OS is more than a chat UI. Without it, the system cannot turn “help me do something” into a visible, controllable, recoverable object.

The second group is Capability services:

1
2
3
4
5

CapabilityRegistryService
AppFunctionIndexer
AgentCardRegistry
McpToolBridge
GuiAutomationBroker

CapabilityRegistryService aggregates all callable capabilities. AppFunctionIndexer handles Android AppFunctions. AgentCardRegistry handles local and remote Agent declarations. McpToolBridge connects tool services. GuiAutomationBroker manages temporary authorization and execution boundaries for GUI automation.

This group decides whether the system can find executable capability from a user goal. Without a unified index, the Agent can only guess with the model, and will fall back to unstable screen tapping.

The third group is Context and Memory services:

1
2
3
4
5

ContextSnapshotService
SemanticIndexService
PreferenceMemoryService
EphemeralTaskMemory
MemoryDeletionService

ContextSnapshotService provides the current scene’s minimum semantic snapshot. SemanticIndexService manages local indexes. PreferenceMemoryService saves long-term preferences. EphemeralTaskMemory saves intermediate state while tasks run. MemoryDeletionService provides visible deletion and verifiable deletion.

This group decides whether the Agent understands the user, and whether the user is willing to be understood. Once memory becomes a black box, trust drops quickly.

The fourth group is Policy and Security services:

1
2
3
4
5
6

AgentPolicyService
ConsentManager
AuditLogService
RiskClassifier
EnterprisePolicyAdapter
SecureExecutionBroker

AgentPolicyService handles data classification and execution rules. ConsentManager manages each confirmation. AuditLogService records calls and side effects. RiskClassifier judges action risk. EnterprisePolicyAdapter connects to company MDM or compliance policy. SecureExecutionBroker places payment, credentials, signing, Passkey, and high-sensitivity actions inside a security boundary.

This group decides whether the system can take responsibility. Without it, every new Agent permission is another incident entry point.

These four service groups can coexist with macro modules such as Agent scheduler, Memory manager, Context manager, and Access manager. Macro architecture helps people understand layers. Engineering decomposition explains what services must actually be added to the system. Agent OS system services are a set of task-, capability-, context-, and policy-coordinated services. They should not be reduced to four big boxes.

Judgment on the Kernel Layer

Architecture diagrams can include wording such as “Linux kernel / Agent OS kernel support,” but the language must be careful.

Agent OS does not need an “Agent kernel” in the short term. It needs existing Linux/Android kernel capabilities to be used better: cgroup for resource control, SELinux for access constraints, seccomp for syscall restriction, Binder for local IPC, mmap CoW for sharing model weights, mmap + madvise or io_uring for on-demand weight loading, userfaultfd and similar mechanisms for ART GC and finer-grained memory management, and TEE drivers for secure hardware.

Agent semantics should not be pushed into the kernel. The kernel should answer resource, security, isolation, and performance questions. What a Task is, what an Artifact is, where user confirmation happens, and which data can go to the cloud should be handled in Framework, system services, and cloud policy.

Android vendors should make this clear early. Do not heavily modify the kernel mainline just for “Agent OS.” In the short term, the more valuable investment is System Server, HAL runtime, on-device model service, SystemUI, permission UI, and cloud protocols. Kernel optimization should serve model-weight sharing, low-power residency, sandbox isolation, and secure execution, not business semantics.

4. OpenAI’s Fork: Android-based or Not Android-based

After clarifying the Android route, we can return to OpenAI. We should not assume OpenAI must use Android.

If OpenAI does not build on Android, it gets a cleaner design space. The home screen can be a task stream from day one. Permissions can be designed around Agent tasks from day one. Apps are no longer the default center. Cloud long-term memory and subscriptions can be deeply bound. It does not have to compromise with Android history or force every function into the old Activity/Intent/Service/Broadcast model.

But it immediately faces a harder question: where do the services users need every day come from?

The difficulty of phone OS extends from kernel, UI, drivers, and system APIs into the app and service network. WeChat, WhatsApp, Instagram, TikTok, YouTube, Uber, DoorDash, Google Maps, Alipay, Meituan, banks, company MDM, carrier services, government apps, school apps, access-control apps, and medical apps are also real-life entry points.

A non-Android OpenAI phone that wants to become a primary device must choose several compensation paths.

First, use Web/PWA. Many services can be completed through the web. A cloud Agent can operate a browser, while the phone provides confirmation and credentials. But web still lags native apps in payment, notifications, offline use, performance, sensors, background execution, and local capability. Top platforms may also restrict automated access.

Second, use cloud app proxies. The Agent opens webpages, calls APIs, runs browsers, or simulates terminals in the cloud, while the phone only handles task entry and confirmation. This fits research, office work, shopping comparison, content processing, and lightweight services. It does not fit high-sensitivity payment, IM, banking, games, ride-hailing live location, or local device control.

Third, use an Android compatibility layer. The foreground is not Android, but it can run Android apps. This preserves service coverage, but the engineering difficulty and legal/commercial negotiation are heavy. App vendors may also restrict the environment through device certification, integrity checks, and risk-control strategies.

Fourth, start as a companion device. Let the OpenAI device be the task entry, voice entry, memory entry, and cloud Agent entry, while relying on the user’s existing iPhone/Android for sensitive actions. This avoids the pressure of replacing the primary device, but it raises the question: why not just use the ChatGPT app on the phone?

Fifth, target enterprise or high-end narrow scenarios. For example, an AI work phone for frequent business travelers, creators, developers, salespeople, doctors, lawyers, and researchers. It does not need to cover all consumer services immediately. If it wins high-value tasks, it can validate the Agent OS product form.

So the non-Android route is feasible, but it must accept a reality: without app compatibility, it is hard to become the first phone; with app compatibility, the old app model drags it back. If OpenAI wants a new OS, the more realistic path is to make task stream, cloud Agent, subscription, and a few high-value services work deeply first, then gradually expand callable capabilities.

From a system architecture perspective, the non-Android route still cannot avoid similar modules:

1
2
3
4
5
6
7
8
9

Task Stream Shell
  ↓
Agent Runtime / Policy / Memory / Capability Registry
  ↓
App Compatibility Layer / Web Runtime / Cloud Tool Runtime
  ↓
Linux or custom kernel primitives
  ↓
SoC, secure element, sensors, modem, power management

It may not be called Android, but the underlying principles remain similar. An OS must manage identity, permissions, processes, storage, network, display, input, sensors, security, and developer interfaces. Agent OS raises “tasks” and “capability calls” to a higher priority. It does not eliminate the basics.

In other words, the Android versus non-Android difference is not “whether there is an Agent.” It is “how to carry the old world.” The Android route carries the old world through compatibility and adds task stream in the foreground. The non-Android route defines a new world through task stream, then fills old-world gaps with web, cloud, and compatibility layers.

The former is steadier. The latter is cleaner. The former is easier to evolve gradually. The latter is more likely to produce a paradigm sample. OpenAI’s advantage is exactly that it has no App Store, no Launcher, and no OEM legacy baggage, so it can dare to put task stream on the first screen. Its weakness also comes from here: it has none of that baggage, and none of those assets.

Three Prototype Forms for a Non-Android Route

If OpenAI really does not use Android, it probably will not start with a “complete phone OS.” Three prototype types are more likely.

The first is a task-terminal phone.

It has a screen, cellular network, camera, and voice entry. It may make calls and receive SMS, but the first screen is not an app grid at all. Its main interface is a ChatGPT Task Stream: pending tasks, running tasks, tasks waiting for confirmation, completed Artifacts. App compatibility depends on web, cloud browser, and a few native services. It looks like a phone, but does not try to fully copy an Android phone.

This device fits high-end users and high-value tasks: business travel, research writing, creative editing, sales follow-up, personal assistant work, developer workflows. It may not cover every life scenario, but it can establish a sample around “reducing cross-app operations.”

The second is a phone companion device.

It does not fight to be the first phone. It binds to the user’s existing iPhone or Android. It handles voice, memory, task initiation, cloud Agent, photo understanding, and lightweight confirmation. Actions involving payment, IM, banking, or company MDM still return to the primary phone for confirmation.

The advantage is lower entry cost and no need to solve app compatibility all at once. The disadvantage is the obvious user question: why not use the ChatGPT app on the phone? So it must be clearly better in always-on presence, low interruption, capture, and task continuity.

The third is an enterprise/professional device.

OpenAI can first target high-value professions with an Agent OS device: doctors checking information and writing records, lawyers organizing cases and contracts, salespeople following up customers, researchers collecting information, journalists organizing interviews, creators processing material, developers handling code tasks. These devices do not need every consumer app to be compatible. They need deep cooperation with professional tools, enterprise systems, browsers, and filesystems.

This route sounds less like a mass-market phone, but it fits the early economics of Agent OS better. How much ordinary users will pay for “opening fewer apps” is uncertain. How much professional users will pay for “saving one hour of high-value work every day” is easier to calculate.

The Cost of OpenAI Not Using Android

The cost of a non-Android route can be split into six categories.

First is app availability. Without Android/iOS native apps, many services can only use web or cloud proxies. The hardest things to replace in daily life are apps with payment, risk control, messaging, location, push, and local device capability, not webpages that a browser can open.

Second is accounts and identity. A phone is an identity container. Passkeys, SIM/eSIM, carriers, bank certificates, company MDM, government services, and family sharing all require long-term accumulation and certification. A new OS must rebuild these relationships.

Third is developer motivation. Why would developers expose capabilities to OpenAI Agent OS? For new users, revenue share, lower support costs, or because users will abandon apps that do not integrate? Before installed base exists, this is hard to answer.

Fourth is platform responsibility. When an Agent submits an order, sends a message, publishes content, or handles a bill on behalf of the user, who is responsible when something goes wrong: user, OpenAI, service app, or merchant? Traditional OS can say “the app is responsible.” Agent OS cannot cut responsibility that easily.

Fifth is supply chain and support. Phones are low-tolerance hardware products. Baseband, antenna, RF, thermal design, battery life, drop resistance, repair, spare parts, regional certification, and carrier testing are not AI-company strengths. System manufacturing partners such as Luxshare can help, but product responsibility remains with the brand.

Sixth is user migration. Users will not abandon iMessage, WeChat history, photo libraries, wallets, health data, membership benefits, and car keys just because of a new entry point. OpenAI must provide a path of coexistence first, migration later.

So the non-Android route is better suited to defining a sample than becoming the global primary phone on day one. It can prove the value of task stream and cloud Agent, then force Android and iOS to absorb that form.

If OpenAI Uses Android, What Must It Solve?

Using Android is not easy either.

First, it must decide its relationship with Google. Does it use GMS? Does it preload Play Store? How does the default assistant compete with Gemini? Can AppFunctions, AICore, and Android AI APIs be deeply used by OpenAI? Without GMS, overseas app availability is affected. With GMS, system entry is constrained by platform rules.

Second, it must decide the position of Android apps inside the task stream. Does it keep a traditional Launcher, or put Launcher as a secondary entry? Does it always let users return to the app grid, or aggressively push Task Stream? The first generation probably must keep a fallback; otherwise users will feel unsafe at critical moments.

Third, it must solve permission credibility. If an OpenAI system assistant asks to read screen, notifications, calendar, location, photos, and messages, users will be highly sensitive. It must provide clearer explanations than ordinary Android permissions: which data is processed on device, which goes to the cloud, which is used only for this task, which will be remembered, and which will never be remembered.

Fourth, it must negotiate with top apps. Without structured capabilities, an OpenAI phone can only rely on GUI Agent to tap screens. If it relies on GUI Agent to tap screens, top apps will restrict it quickly. OpenAI must give service providers a reason: letting the Agent call you brings more transactions, less support cost, higher conversion, and divisible responsibility.

Fifth, it must manage hardware and model power consumption. If a ChatGPT-level experience continuously uses network, NPU, microphone, camera, and screen, users will feel heat and battery drain before they feel intelligence. The first hard metric of an Agent phone may become “after a full day always on, does the user still want to keep it enabled?” Model ranking is only part of the answer.

This is the difficulty of the Android route: it can become a phone faster, but is more easily dragged back by the old phone paradigm. A non-Android route can look more like Agent OS, but is harder to make into a phone.

5. Device-Cloud, Hardware, Industry, and Timeline

Doubao phone and device-cloud combinations such as OpenClaw/ArkClaw provide a useful observation point: AI phones need device and cloud to work together. They cannot stop at on-device model plus cloud chat.

The device-side role is not to run a small model for show. The device must do five things.

First, real-time sensing. Screen, notifications, location, camera, microphone, Bluetooth, sensors, foreground Activity, clipboard, and system settings are only known on device.

Second, low-latency judgment. Wake word, intent classification, risk judgment, simple summaries, on-device embeddings, and on-device VLM preprocessing require low power and low latency. They are not suitable for cloud every time.

Third, privacy closure. Payment credentials, biometrics, health data, private IM content, sensitive photos, precise location, and company-managed data cannot be freely read by the cloud. The device needs private compute areas and minimal context trimming.

Fourth, final confirmation. The Agent can suggest and prepare, but money, messages, orders, deletion, publication, and authorization often need user confirmation on device. Confirmation carries responsibility boundaries and cannot be treated as ordinary UI detail.

Fifth, interruption and braking. Users must be able to stop tasks, revoke grants, view history, and delete memory at any time. This control should be held by the device-side system, not hidden inside a cloud conversation.

The cloud’s role is not just “a larger model” either. The cloud must carry long tasks that the device should not bear.

First, long-running execution. Booking travel, organizing research, reviewing contracts, writing reports, editing videos, running code, comparing prices, and contacting multiple services are not one-request tasks. They may last minutes, hours, or days.

Second, tool environments. Browsers, code sandboxes, filesystems, crawlers, document processing, spreadsheets, image generation, and audio/video processing can be started from device, but should not all be hosted there.

Third, long-term memory. User preferences, style, common routes, work habits, family relationships, and project context need to accumulate across devices, sessions, and years. The device can store indexes and sensitive truths. The cloud can store memory graphs filtered by policy.

Fourth, Agent collaboration. Complex tasks split into sub-tasks: research, comparison, execution, verification, writing, scheduling, notification. The cloud is better suited to running multiple isolated sub-Agents at the same time.

Fifth, cross-device continuity. The user starts a task on phone, continues editing on computer, hears progress in earbuds, confirms a route in the car. The Agent should bind to user identity and task state, not a single device.

The device-cloud boundary needs a protocol layer containing task state, messages, Artifacts, permissions, data classification, push callbacks, failure recovery, and audit identifiers. Plain HTTPS APIs are not enough. A2A’s Task, Message, Part, Artifact, streaming, and push notification designs can be references. Android local execution may not use HTTP A2A, but these abstractions fit Binder or internal system services well.

From device-cloud division of labor, future Agent-phone SoC design will also change. In the past, phone chips emphasized CPU/GPU/NPU peaks, ISP, baseband, and power. Agent phones add several persistent pressures:

• Low-power NPU use from continuous context understanding.
• Memory pressure from multi-task state, embeddings, KV cache, and indexes.
• Always-on scheduling of camera, microphone, sensors, and model preprocessing.
• TEE/StrongBox/secure element hosting of credentials, payments, and private memory.
• Network, push, resume, and background wakeups from device-cloud round trips.

This explains why OpenAI, if serious about Agent phones, would care about processor co-design rather than only buying an off-the-shelf flagship SoC. AI Agent phone processors need to reallocate budget around continuous context, memory tiers, low-power sensing, and secure execution. They cannot be understood only as “phone chips with a larger NPU.”

Data Classification Decides Device-Cloud Routing

Device-cloud division should not be decided only by model size. A more reasonable method is to decide jointly by data sensitivity, real-time needs, task duration, and side-effect risk.

Phone data can be roughly split into five layers.

L0 is public data. Weather, public webpages, public product information, and public traffic information can be processed directly in the cloud.

L1 is low-sensitivity personal data. For example, a user-provided requirement, current city, preferred hotel type, or public calendar title. This can go to the cloud after authorization, but should have a purpose and expiry.

L2 is medium-sensitivity personal data. For example, full calendar, shopping history, common contacts, location history, file summaries, and work project names. This can go to the cloud, but should be trimmed, desensitized, or temporarily authorized by task.

L3 is high-sensitivity data. For example, chat content, original photos, health data, precise location, company files, family relationships, and financial information. This should be processed on device by default. Only when the user explicitly confirms and the task requires it should the minimum slice be sent.

L4 is data that must not leave the device. For example, biometrics, payment keys, Passkey private keys, bank verification codes, enterprise certificates, and device keys. This data can only be used inside the device security boundary. The cloud should receive at most “user confirmed” or “signature completed.”

Device-cloud routing should work like this:

1
2
3
4
5
6
7
8
9
10
11

User task
  ↓
Identify task type and risk
  ↓
Split required data
  ↓
Classify by L0-L4
  ↓
On-device processing / desensitized cloud / one-time grant / cloud forbidden
  ↓
Record audit and expiry

This mechanism sounds tedious, but it is the prerequisite for trust. Users already feel sensitive when authorizing an app to access the photo library. An Agent phone reads much more context. Without data classification, it becomes more black-box-like than any app.

Android vendors have an opportunity here: make data boundaries visible in system UI. For example, a task card can clearly show:

• This task used 2 images, 1 calendar event, and 3 public webpages.
• What went to the cloud was invoice fields, not the original album.
• Payment credentials did not leave the device.
• Temporary context will be deleted 24 hours after task completion.
• The user can delete Artifacts and memory in task history.

This UI does not have to be flashy, but it is more persuasive than a slogan about privacy.

Cloud Agent Is Not a Stateless API

Many phone teams are used to treating the cloud as an API: the device sends a request, the cloud returns a result. That model does not fit Agent phones.

Complex tasks need the cloud to keep state. When the user asks the Agent to research travel plans, the cloud opens webpages, records candidate hotels, compares flights, saves a budget table, waits for price changes, and generates itinerary documents. These intermediate states cannot be stuffed back into the prompt every time, and they cannot restart every time.

A cloud Agent is more like a user-specific workspace. It can be a combination of container, microVM, browser session, filesystem, vector database, task queue, and tool set. Each user’s task needs isolation. Each task’s files and credentials need isolation. When the task ends, cleanup must be possible.

This is where cloud Agent approaches such as OpenClaw/ArkClaw have value: the cloud gives the user a long-running execution environment, not just a larger model for the phone. It can run browsers, code, documents, spreadsheets, and image tools; it can continue while the user is offline; it can push back to the phone when confirmation is needed.

But this cloud shape also has three risks.

First is cost. An always-on or semi-always-on cloud Agent is much more expensive than ordinary API calls. Subscription, hardware bundling, task quotas, and enterprise payment will become part of the business model.

Second is security. The more tools a cloud Agent gets, the more it needs sandbox isolation, network access control, file access control, credential minimization, and prompt-injection defense. It cannot leak files or place orders for the user just because it read malicious instructions on a webpage.

Third is explainability. Users should not only know that “the cloud handled it for you.” They need to see task steps, call sources, generation basis, and final confirmation points. The longer the task, the less comfortable users become without explanation.

So device-cloud architecture cannot be simplified to “small device, big cloud.” The device handles scene, privacy, confirmation, and braking. The cloud handles long tasks, tools, memory, and concurrency. The protocol layer manages state, authorization, Artifacts, and audit.

Do Not Overstate or Understate the On-device Model

On-device models will not replace cloud models in the short term. Power, memory, thermal design, and cost constraints on phones mean they are not suitable for all complex reasoning. But on-device models are not supporting actors either.

They are best suited to five kinds of work.

First is wake and intent classification. When a user sentence arrives, the device first judges whether cloud is needed, whether privacy is involved, and whether the task can be handled locally.

Second is context trimming. The device trims screen, notifications, calendar, photos, and files down to the minimum slices needed by the task, then decides what can go to the cloud.

Third is privacy-sensitive understanding. Fraud-call detection, verification-code recognition, health anomaly prompts, private photo classification, and company-file summaries can be done or preprocessed on device.

Fourth is fast feedback. Translation, summaries, rewriting, simple Q&A, local image editing, and speech-to-text are latency-sensitive. On-device models have an advantage.

Fifth is failure fallback. When the network is down, weak, cloud-congested, or cloud permission is disabled, the device must provide a minimum usable capability.

This is also the value of processor co-design. Agent phones need not only high compute, but also low-power always-on models, fast cold start, model-weight sharing, embedding indexes, KV/cache management, and secure memory. NPU peak numbers only explain part of the experience. Continuous operation determines whether users keep the Agent enabled.

The Different Positions of Five Groups of Players

OpenAI’s position is the most special. It has models, consumer brand, subscription relationship, and global developer attention, with no traditional phone baggage. It can aggressively put Task Stream on the first screen, treat apps as background capabilities, bind subscription to hardware, and differentiate through cloud long tasks. Its problems are also clear: lack of local services, payments, maps, social networks, and channels; China almost certainly requires local partners; callable rights for apps need to be renegotiated; first-generation hardware reliability pressure is high.

Apple’s advantages are on-device hardware, system permissions, privacy brand, App Store, App Intents, and user trust. It does not need to radically change the home screen to weave Siri, Spotlight, notifications, Shortcuts, App Intents, and Private Cloud Compute together. Apple’s constraint is that it does not want to hurt the App Store or existing usage habits. It will be slow, but slow does not mean weak. If Apple makes task suggestions, App Intents, and system confirmation work well, user migration cost is low.

Google + Samsung is the standardization path for the Android camp. Google has Android, Gemini, AppFunctions, A2A, and cloud. Samsung has high-end hardware, Galaxy AI, Now Brief, Bixby, Now Bar, and global users. Samsung’s willingness to hand system entry to multi-Agent combinations makes it an important test bed for Android. The difficulty is dispersed interests: Google, Samsung, Qualcomm, MediaTek, OEMs, carriers, top apps, and regional regulators all influence system choices.

ByteDance Doubao is one of the most valuable samples in China. Doubao phone, system-level permissions, Virtual Display, UI-TARS, OpenClaw/ArkClaw internal directions, and the public-facing Coze Space-style cloud Agent environment push “AI operates apps for users” from slides to devices. Its value is not sales, but showing two things early: Agent phones are feasible, and third-party apps will push back. ByteDance has Chinese content, video creation, office tools, models, advertising, and e-commerce services, but no dominant mainstream phone OS. To move forward, it must go from “can tap screens” to “protocol calls, auditable authorization, and commercial distribution.”

Huawei, Xiaomi, OPPO, vivo, and Honor have system permissions, hardware, channels, local services, and compliance capability. They are well positioned to build a Chinese version of Agent OS, but are also most likely to build separate systems. If every vendor has its own skill protocol, memory format, AgentCard, payment confirmation, and task state, developers will relive adaptation hell. The Chinese Android camp should not rely only on one more AI assistant inside the ROM. It needs a relatively stable common language for capability declaration, permission authorization, task state, data classification, and audit records.

Another group cannot be ignored: top apps. They may not want to be called by Agents. For WeChat, Taobao, Alipay, Meituan, Douyin, banks, and games, the foreground entry point is a business asset. If an Agent takes the entry point away, platforms will worry that user relationships, ads, transaction share, risk control, and responsibility are all rewritten. In the next few years, the largest negotiation in Agent OS will happen among system vendors, AI vendors, and top apps. The question is how entry, data, and responsibility are redistributed.

The Real Constraints of the Five Groups

To avoid turning each company’s path into slogans, place them in one constraint table.

In this table, OpenAI’s advantage is the opposite of other players. Traditional phone vendors have phones but lack a strong enough Agent entry point. OpenAI has the Agent entry point but lacks the most troublesome service relationships in the phone world. It is moving from task entry toward hardware. Other vendors are moving from hardware and system permissions toward task entry.

The two sides will meet around 2028.

If OpenAI does well enough, it proves that “task stream first” is feasible. Traditional vendors will absorb its interaction and task model.

If OpenAI hardware fails, it still leaves boundary experience: which tasks users are willing to give to an Agent, which services must return to apps, which permissions users cannot accept, and which hardware forms do not fit always-on operation.

That is its disruptive value. It does not have to become the largest phone vendor to change the phone industry.

Why Supply-chain Signals Are Worth Watching

Kuo’s report puts MediaTek, Qualcomm, and Luxshare together. That is already beyond ordinary supply-chain gossip. If it is true, OpenAI’s target covers two layers: finding manufacturing partners, and participating in system design, SoC definition, and manufacturing coordination.

Agent phones will change SoC requirements.

Traditional flagship SoCs emphasize peaks: CPU single-core and multi-core, GPU, NPU TOPS, ISP, baseband. Agent phones need these too, but they care more about sustained load.

An always-on Agent will not run a big model once every ten minutes and stop. It will continuously listen for low-power events, maintain small context, frequently run embedding and rerank, occasionally call a VLM to inspect screen or camera, sync cloud tasks in the background, wait for push, and bring confirmation cards to the foreground.

This shifts pressure to several places.

First is low-power sensing. Voice, movement, location, Bluetooth, calendar, and notification events must be processed continuously without waking big cores and big models every time. Sensor hub, DSP, small NPU, and always-on island become more important.

Second is memory hierarchy. Model weights, KV cache, embedding indexes, task state, and screen semantic snapshots compete with traditional apps for memory. The difference among 8GB, 12GB, and 16GB will show up in background retention and whether multiple Agent sub-tasks can be carried at once.

Third is secure storage. Agents touch credential references, payment confirmation, private preferences, company materials, and identity tokens. StrongBox, TEE, secure elements, encrypted storage, and remote attestation move from “security specs” into user experience.

Fourth is network and push. Device-cloud tasks round-trip more often than ordinary app requests and depend more on reliable push. Task interruption, retry, weak-network recovery, roaming, and airplane-mode recovery all need design.

Fifth is heat and battery. Users will not accept a phone that heats up and drains visibly because AI is always on. Agent phone success may be decided by a plain question: if enabled by default all day, is battery life still trustworthy?

MediaTek and Qualcomm each have a position here. Qualcomm is strong in high-end Android, baseband, NPU, and global flagship cooperation. MediaTek is strong in cost, integration, and flexible cooperation with some OEMs. OpenAI contacting both can be reasonably understood as comparing different SoC definition paths, not necessarily having locked one traditional phone plan.

Luxshare’s role is also worth watching. It is not just contract manufacturing. If an AI Agent phone rewrites hardware interaction, it may involve new buttons, microphone arrays, sensor layout, thermal design, antenna space, secure-element placement, and always-on display strategy. System co-design matters more than ordinary ODM work.

But judgment should stay restrained. A supply-chain report being true does not mean the product will succeed. In hardware, prototype, EVT, DVT, PVT, mass production, channels, and support can all change direction. For Android practitioners, the more valuable observation is how OpenAI defines Agent phone SoC, permissions, entry point, and device-cloud division, rather than betting on phone sales.

2026 to 2032: What Is More Likely to Happen

2026 to 2027 is the stage of “assistant moves forward, app grid remains.”

In this stage, vendors will hand long-press side buttons, voice, earbuds, lock screen, notifications, and floating entry points to AI assistants. Now Brief, Now Bar, system briefings, smart notifications, cross-app summaries, screen Q&A, on-device image editing, live translation, and meeting minutes will keep spreading. GUI Agents will appear in some tasks. AppFunctions / App Intents will expand from top scenarios.

The experience will be mixed. Many features will look like AI phones, but are still old OS plus assistant. Users will feel that “good tasks are amazing, bad tasks are awkward.” This stage first exposes four engineering gaps: task state, permission confirmation, capability routing, and failure recovery.

2028 may be the window for an external reference.

If an OpenAI phone enters mass production as Kuo reported, it may not immediately take mainstream share from iPhone or Galaxy. But it can provide an industry reference: can the home screen not start from apps? Can tasks become first-class OS objects? Can hardware subscription merge with cloud Agent? Can phone SoCs be designed around continuous context? Can users accept a more proactive task stream?

Even if OpenAI’s first phone has limited sales, it may play the defining-sample role that early Surface, Pixel, or Vision Pro played. It can expand review metrics from “camera, battery, screen, benchmark” to “task completion rate, confirmation design, privacy classification, long-task recovery, cross-device continuity, app callability.”

From 2029 to 2030, app entry points will begin to retreat more visibly into the background.

This does not mean app icons disappear. What is more likely is that high-frequency user operations are gradually absorbed by task streams. The home screen may have three persistent views: conversation, task, memory. Notification center becomes a task status panel. App Store gains a capability-store attribute. Developers submit not only apps, but callable functions, schemas, pricing, permissions, and result formats.

At that point, phone vendors will find that the Launcher problem has expanded from desktop layout into a task operating system. Whoever better understands user state, schedules capabilities, and controls risk owns the next entry layer.

From 2031 to 2032, multi-device Agents will begin to exceed single-phone Agents.

The phone remains central, but it is no longer the only entry. Glasses handle vision, earbuds handle voice, watches/rings handle physiological signals, cars handle travel, computers handle complex editing, TVs handle family content, and home devices handle spatial state. The phone becomes the identity, authorization, confirmation, and high-compute local node.

At that stage, the Agent OS object is no longer one device, but the user’s device group. A task starts on the phone, continues on the computer, reminds in earbuds, confirms a route in the car, and announces through the home speaker. The OS boundary expands from “device operating system” to “personal task system.”

Use Real Task Types to Test the Timeline

A timeline written only by year looks like a prediction list. A better way is to watch when different task types become stable.

The first type is information organization: meeting minutes, webpage summaries, notification organization, email drafts, travel plan comparison, information collection. These tasks do not depend heavily on high-risk actions or deep top-app interfaces. They will mature quickly in 2026-2027. They will be the first reasons users accept Agent phones.

The second type is light execution: creating calendar events, generating to-dos, saving files, sending to self, organizing albums, setting reminders, simple shopping comparison. These need system permissions and some app cooperation, but risk is low. They fit AppFunctions, App Intents, and system capabilities, and will enter high frequency around 2027.

The third type is transaction tasks: booking tickets, booking hotels, ordering food, hailing rides, buying products, canceling subscriptions, reimbursement, bill payment. These involve payment, price, after-sales, responsibility, and risk control. GUI Agent can demonstrate them first, but stable execution needs platform cooperation and structured capability. Large-scale stability may come after 2028.

The fourth type is social and content publishing: sending WeChat messages, replying to emails, posting to Weibo, Xiaohongshu, or Douyin, commenting, direct messaging. The risk here is not only technical, but also tone, relationships, and consequences. Agents can draft and suggest, but direct sending will be cautious. Before trust is built, confirmation is hard to remove.

The fifth type is high-sensitivity tasks: banking, securities, insurance, healthcare, government services, company approval, contract signing, identity authentication. These will open latest and best test Agent OS permission and audit design. Without strong confirmation, strong audit, and strong responsibility boundaries, platforms will not let them in.

The development speed of these five task types is more valuable than the question “which year will Agent OS appear?”

From 2026 to 2027, information organization and light execution mature.

From 2028 to 2030, transaction tasks begin to be stably absorbed by system-level Agents.

After 2030, social delegation and high-sensitivity tasks may open within specific boundaries.

This also explains why even if OpenAI’s first phone exists, it will not replace all apps at the beginning. It is more likely to make information organization, long tasks, research, creation, and light execution strong first, then gradually touch transactions and high-sensitivity tasks.

6. Product Shape, Market Split, and Responsibility Distribution

If the Agent OS phone from 2028 to 2032 is compressed into one product shape, it probably will not be a full-screen chatbot.

Chat must remain because natural language is a low-cost entry. But chat is not the destination. Users do not want to talk to the phone all day. They want the phone to interrupt less, switch less, forget less, and ask for repeated input less.

The more likely first screen is a task stream.

In the morning, the phone shows tasks that need confirmation today: flight-change suggestions, meeting-material summaries, children’s school materials, bill due reminders, fitness plan, and research results completed by the cloud last night. Each task has state, source, next step, risk, and confirmation button.

During the day, the user says, “Mark the contract risks in this PDF and give me a shareable version before 3 p.m.” The phone sends the file to a cloud sandbox, keeps sensitive-field indexes on device, and the cloud generates the report. When complete, it pushes back a task card. After user confirmation, the system calls an office app to send it.

At night, the user says, “Take my parents to Yunnan next month, do not make it tiring, budget 15,000.” The cloud runs routes and hotels. The device reads family calendar and weather preferences. Hours later, the system provides three plans. The user does not need to stay in the chat window.

Behind these experiences are three UI levels.

The first is Task Stream. It combines home screen, notifications, and lock screen, managing tasks that wait for confirmation, are running, are completed, or failed.

The second is Capability View. Users or developers can view which apps, services, and Agents can be called, what they can do, what permission range they have, and what price and risk they carry.

The third is Memory View. Users can see preferences, projects, relationships, common information the Agent remembers, and can edit or delete them.

These three views will gradually replace part of app-grid time. The app grid remains, just as filesystems and command lines remain. It is simply no longer the starting point for most users to complete tasks.

What the First Screen May Become

The first screen is unlikely to be only one big chat box. A pure chat entry has two problems: users must constantly express themselves, and task state is invisible.

A more reasonable first screen may have four blocks.

The first block is current tasks. It shows running, waiting-for-confirmation, and near-deadline tasks. Each task card has source, state, next step, and risk prompt.

The second block is today’s state. It summarizes calendar, weather, travel, health, messages, bills, and family matters without filling the screen with everything.

The third block is quick expression. Users can create new tasks through voice, text, screenshot, photo, or shared content.

The fourth block is app fallback. Users can still open the traditional app grid, especially when the Agent is unreliable, when the user wants to operate manually, or when the task is high-risk.

The ratio among these four blocks will vary by user. Heavy users rely more on task stream. Conservative users still center on the app grid. The system should not force everyone into Agent mode immediately.

The Position of Generative UI

Future phone interfaces will not choose only between app UI and chat UI. Generative UI will become part of task results.

When the user says, “Give me three family-trip plans for Yunnan,” the system should not only return a paragraph or jump to an app. A better result is an interactive panel: budget, route, hotel, transport, elder-friendliness, cancellation policy, weather risk, and options needing confirmation. The user can adjust preferences inside the panel, and the Agent continues execution.

When the user says, “Rewrite this report for my boss,” the system can generate a comparison view, revision suggestions, source references, risk prompts, and one-click export, instead of dumping a new paragraph into chat.

The key to Generative UI is turning intermediate task state into an operable interface, not showing off. This lets users participate in decisions instead of handing all judgment to the model.

For Android, Generative UI raises new questions for View/Compose/WebView: where does the UI come from, who is responsible for security, can it call system capabilities, can it embed third-party services, can it be screenshot-audited, and can it recover offline? Eventually it still returns to Task, Capability, Policy, and Artifact.

Memory View Becomes the Trust Entry

The smarter an AI phone becomes, the more users will ask one question: why do you know that?

This makes Memory View crucial. Users need to see what preferences, projects, contact relationships, common addresses, writing style, and travel taboos the Agent remembers. They also need to edit, delete, and pause memory.

Memory UI should not be buried like a privacy list in Settings. It should become one of Agent OS’s common interfaces. Memory is the long-term contract between user and Agent. Backend technology is only one part.

A good memory interface should support:

• Viewing by topic: travel, work, family, health, shopping, writing.
• Viewing by source: from conversation, calendar, files, or manual user input.
• Viewing by sensitivity: ordinary preference, high-sensitivity information, data that must not leave the device.
• Expiry: some preferences are valid only for this trip, project, or month.
• Correction: users can say “this is not my preference, it was a one-time situation.”

Without Memory View, the Agent gradually becomes an unexplained black box. With Memory View, users may be willing to hand it more tasks.

China Will Follow an Independent Path

After product shape, China needs a separate look. The China market will not simply copy OpenAI’s phone route.

First, OpenAI can hardly enter China’s primary-phone market directly. Model compliance, data cross-border rules, app stores, local payment, maps, IM, local-life services, government relations, and carrier relations are each complex enough.

Second, Chinese users’ high-frequency services concentrate in a small number of super apps and local platforms. An Agent that wants to finish tasks must deal with WeChat, Alipay, Taobao, Meituan, Douyin, JD, Didi, banks, and government services. This involves technical, commercial, and compliance problems at the same time.

Third, Chinese Android vendors have stronger ROM customization capability and more complex vendor fragmentation. Each can build its own AI assistant, but without common capability standards, developers and top apps will not want to adapt separately for each vendor.

Fourth, the Doubao route will remain referential. It connects system permissions, Chinese content, cloud Agent, GUI automation, and its own app network, showing that Chinese companies may build a different Agent phone from Apple/Google/OpenAI. It also reminds everyone that the right to be legally called by other apps may be scarcer than model parameters.

The China market is more likely to have multiple forces coexist:

• Huawei follows its own system and services on HarmonyOS NEXT.
• Xiaomi, OPPO, vivo, and Honor strengthen system Agents in Android/AOSP systems.
• ByteDance, Tencent, Alibaba, Meituan, and other service platforms provide their own Agent capabilities and developer tools.
• Top apps defend against GUI automation, but open structured capabilities when commercial returns are clear.
• Domestic protocols borrow from A2A/MCP/AppFunctions while adding local compliance, data classification, real-name, payment, and risk-control fields.

This path will not be as clean as OpenAI’s ideal, but it will fit Chinese service reality better. For Android phone practitioners, the more important question is whether system permissions, on-device models, domestic cloud, top apps, payment confirmation, and task audit can combine into a credible execution system, not whether another ChatGPT is rebuilt.

Three Product Lines in China

China is more likely to have three product lines at the same time, not one Agent OS.

The first is the OEM system-assistant line. Huawei, Xiaomi, OPPO, vivo, and Honor will keep moving assistants into the system layer, connecting photos, notifications, calendar, files, cars, homes, and wearables. Their advantage is system permissions and hardware entry. Their weakness is that model and service coverage may not be strongest.

The second is the service-platform Agent line. ByteDance, Tencent, Alibaba, Meituan, Baidu, JD, and others will package their content, transaction, office, and local-life capabilities into Agents or tools. Their advantage is services and user behavior. Their weakness is that they do not get full system permissions and may not become the default entry.

The third is the enterprise and vertical Agent line. Finance, healthcare, government, education, manufacturing, and retail will have their own controlled Agents. They care more about compliance, data boundaries, and audit, and will not easily integrate with consumer general assistants.

These three lines will cooperate and guard against each other. OEMs want to be the default entry. Service platforms want to protect user relationships. Industry systems want to keep data boundaries. Whether Agent OS becomes large in China depends not only on model level, but also on whether these three lines can form an acceptable cooperation model.

For users, the best experience is a system-level Agent that can call multiple platform services. For platforms, the safest method is to expose controlled capabilities, not all data and UI. For regulators, the most acceptable design is that every high-risk action has identity, authorization, audit, and responsibility. The overlap acceptable to all three parties is the viable area for China’s Agent phone.

What Doubao Phone Reminds the Industry

The biggest value of Doubao phone is not that it “built an AI phone,” but that it exposed a group of hidden problems early.

First, system-level permissions are useful. Without them, an Agent can hardly understand the screen, operate apps, receive notifications, or maintain background tasks. The capability gap between an AI app and a system Agent is large.

Second, GUI Agent is impactful. It lets users see for the first time that “the phone can really operate apps for me.” Even if unstable, it changes imagination.

Third, top apps will push back. Platforms will not unconditionally accept an external Agent entering their transaction, social, and content scenarios. They will restrict it from security, risk control, user experience, and commercial-interest angles.

Fourth, an owned service network matters. ByteDance can connect Doubao, Douyin, Jianying, Feishu, Volcano Engine, content, and ads, so it has more service support for an Agent phone than a pure hardware vendor. If other OEMs rely only on models and system permissions, service depth will be insufficient.

Fifth, user acceptance may be higher than expected, but trust building will be slower than expected. Users will be attracted by “it can help me tap apps,” while also worrying about “what did it see, will it tap randomly, will it leak data?”

These five points matter for all Android vendors. Agent phones will not stay in the smooth execution shown in promotional videos. They will repeatedly balance system permissions, third-party relationships, and user trust.

Several Boundary Judgments

After product and market, the boundaries should be put on the table.

First, Agents will not replace apps all at once. They will replace entry points first, then part of cross-app operations, and only later change app business forms. Apps remain service providers, but must accept being called by system-level Agents.

Second, GUI Agent is a necessary transition, not the long-term answer. It can prove demand and cover long-tail apps, but it should not carry high-risk actions. The long-term answer is structured capability, system-level authorization, and auditable tasks.

Third, on-device privacy becomes a brand differentiator. Users can accept smarter Agents, but not secret learning and secret execution. Which data stays on device, which goes to cloud, who can audit, how deletion works, and how errors are traced must be clear from day one.

Fourth, OpenAI’s hardware value may not come from sales. It is more like an industry pressure source. As long as it produces a daily-usable task stream, Android vendors and Apple will be forced to explain why their OS still starts from an app grid.

Fifth, Android’s opportunity is standardization. If AppFunctions, A2A, MCP, on-device model services, permission systems, task state, and notification entry points form a stable combination, Android does not need each vendor to build a private Agent OS. Conversely, if every vendor privatizes it, developers will escape back to web and top platforms.

Sixth, a non-Android route has a chance, but is hard to become a primary phone directly. It may first appear as a high-end AI device, companion device, or narrow work phone. Only after app compatibility, service coverage, and trust mechanisms are complete can it move into primary-phone position.

Seventh, Agent OS competition expands from models to systems. Models determine the ceiling of understanding and planning. OS determines whether the Agent can see state, call capabilities, obtain authorization, execute actions, recover failures, and take responsibility. A model without OS permission can only wait for the user to bring problems to it. An OS-constrained Agent can keep working inside the user’s life.

Back to OpenAI: Why Its Disruption Matters

Now return to OpenAI building a phone. The point is no longer only whether the hardware rumor is true. It puts industry questions back on the table. If the device is eventually confirmed, there are roughly two outcomes.

One outcome is an AI Agent phone based on Android or Android compatibility. Then it gives the Android camp an assignment: why can an outsider put task stream on the first screen while traditional OEMs still stuff AI features into Settings, Gallery, and assistant apps?

The other outcome is a more Agent OS-like new device without Android. Then it gives the industry another assignment: without app-grid history, how should a phone or portable device organize tasks, memory, permissions, and cloud execution?

Whichever path happens, it makes the phone industry ask the right question.

For more than ten years, phone OS competition has centered on screens, apps, imaging, performance, and services. Over the next few years, competition will gradually move toward: who can manage user state, who can organize tasks, who can make apps reliably expose capabilities, who can draw privacy boundaries between device and cloud, and who can make users trust a system that acts proactively.

From an Android phone practitioner’s perspective, Agent OS will not overturn Android or iOS overnight. More likely, Android and iOS both grow parts of Agent OS. OpenAI, ByteDance, Google, Apple, Samsung, Huawei, Xiaomi, OPPO, vivo, and Honor all start from their own resources and compete for the task entry point.

The risk is that the industry keeps understanding AI phones as a set of feature points. If the industry only builds AI summaries, AI photo editing, AI search, and AI voice assistants, then once OpenAI builds a usable Task Stream, it will push the problem into the system layer.

This pressure will not stay inside OS vendors. Once Task Stream enters the home screen, apps must answer whether they are willing to be called. Service platforms must answer how transactions and attribution are shared. The system must answer who is responsible after mistakes. So the next thing to watch is how developers, business, and responsibility are redistributed.

Developers, Business, and Responsibility Will Be Redistributed

Agent OS changes the relationship between developers and platforms.

In the app era, developers competed for install, open, dwell time, payment, and return visits. App icon was the entry. Push was recall. App stores were distribution. Ads and subscriptions were monetization.

In the Agent OS era, developers must compete for a new thing: being selected by the Agent.

When the user says “order a bouquet for me,” which service does the system choose? By price, distance, reputation, fulfillment, historical preference, platform partnership, commission, or the user’s past habit? This selection power used to belong to the user and search/app stores. It may move to Agent and system.

This will bring new developer interfaces.

Apps need to declare capabilities: I can order flowers, check inventory, book, cancel, refund, and issue invoices.

Apps need to declare constraints: which cities are available, which actions need login, which actions need user confirmation, and when cancellation is unavailable.

Apps need to declare price and share: how orders brought by Agents are attributed, whether the system or AI platform takes a commission, and whether coupons can be used.

Apps need responsibility interfaces: who is responsible for order failure, how refunds are handled, where customer support entry is, and how audit logs connect.

This is not a pure technical API. It changes the app’s business position.

Top apps will worry about losing entry, so they will require brand display, user confirmation, transaction attribution, and risk-control authority. Smaller apps may be more willing to integrate because Agents can bring new traffic. System vendors will want to build capability stores. AI vendors will want to own the task entry point.

Developer platforms also change. Future developer docs will not only teach UI writing. They will teach capability schemas, callable action definitions, Artifact returns, Agent-induced error handling, sandbox testing, and revenue sharing.

Android AppFunctions is the beginning of this direction. It lets apps expose capabilities to systems and Agents. To support full Agent OS, more complicated business and responsibility layers are needed.

The New Ranking Problem

Agent OS creates a ranking problem similar to search.

When the user says, “Find a nearby restaurant suitable for kids,” which restaurant does the Agent choose? User preference, friend recommendation, platform ranking, ads, commission, or model judgment? If Agent results affect transactions, transparency and fairness appear.

In the search era, users could see multiple links. In the app era, users could open multiple apps and compare. In the Agent era, if the system directly gives one suggestion, the selection process becomes less visible.

This creates new product requirements:

• Mark commercial recommendations.
• Allow users to see candidates.
• Allow users to specify preferences and disable platforms.
• Provide comparison basis for high-value transactions.
• Keep audit records for enterprise and regulatory scenarios.

OpenAI, Google, Apple, Android OEMs, ByteDance, Alibaba, Tencent, and Meituan will all face pressure here. Agents save users time, but cannot turn user choice into invisible commercial allocation.

Responsibility Spreads from Apps to the System

When a traditional app fails, responsibility is relatively clear. If food delivery is wrong, find the food-delivery platform. If payment fails, find the payment platform. If navigation is wrong, find the map. The OS mostly provides capabilities and security boundaries.

When Agent OS fails, responsibility becomes more complicated.

If the Agent misunderstands the user and books the wrong hotel, is the model responsible or the user?

If the Agent understands correctly but calls a more expensive service, is the system recommendation responsible or the service platform?

If GUI automation taps the wrong button, is the executor responsible, the app UI change responsible, or the system responsible for missing second confirmation?

If a cloud Agent is induced by webpage prompt injection and leaks user files, is the cloud sandbox, model, webpage, or user authorization responsible?

These questions force system design. Not every action can be automatic. Not all data can enter the model. Not every result can be submitted directly. Agent OS must treat “high-risk actions require confirmation” as a base rule, not a post-launch popup.

High-risk actions will keep user confirmation for a long time. Payment, messaging, publishing content, deleting data, signing contracts, submitting government services, financial trading, and medical advice will not become fully automatic quickly. The Agent can prepare, compare, fill forms, and explain, but the last step should return control to the user.

Keeping confirmation rights for high-risk actions with the user is a prerequisite for sustainable Agent OS.

Observations from an Android Phone Practitioner’s Perspective

This section does not need to become an action plan. Professional teams already understand side-button entry, permission UI, on-device models, cloud tasks, and developer interfaces. The more valuable move is to narrow the observation angle: in the next few years, the Android camp’s test point will shift from “is there an AI assistant” to whether AI can enter the system object model.

The first observation point is Task. If tasks still exist only inside chat history, the Agent cannot really enter the system layer. Only when tasks have state, progress, Artifacts, cancellation, recovery, and history can they enter home screen, notifications, lock screen, car, and watch.

The second observation point is Capability. GUI Agent will continue to exist, but it is more like a compatibility path. Stable experience depends on whether AppFunctions, AgentCard, MCP tools, system-private capabilities, and top-app partnership interfaces can be uniformly indexed, routed, and audited.

The third observation point is permission and memory. The stronger an Agent phone becomes, the more it must explain “what data this task used, which data went to the cloud, which stayed on device, which will be remembered, and which can be deleted.” This must land in system UI and audit records, not stay as a privacy slogan.

The fourth observation point is device-cloud state. If a cloud Agent is only a stateless chat API, it cannot carry long tasks. The device cannot be only an entry point either. It must handle scene, confirmation, braking, and sensitive-data boundaries. What syncs between device and cloud should be tasks, Artifacts, authorizations, and audit, not an ever-growing chat context.

If an OpenAI phone appears, its meaning for Android may be here: it may not answer every question well, but it will make the industry see the importance of these system objects again. For practitioners, this section is a prompt: do not only look at how many AI features a launch event adds. Look at whether task, capability, permission, memory, and device-cloud state become OS-managed objects.

Final Judgment

Return to the five things in the introduction: state understanding, capability providers, task runtime, device-cloud division, and cross-boundary execution. Compress the whole article into six sentences.

First, phones will not be replaced by Agents. Phones will become the Agent’s most important state node.

Second, apps will not disappear, but their foreground entry role will gradually decline, while capability interfaces gain weight.

Third, Android’s short-term path keeps the existing system and adds a task system across Framework/SystemUI/permissions/notifications/on-device models/cloud protocols.

Fourth, OpenAI may or may not build on Android. Without Android, it has more freedom but a harder time carrying app services. With Android, it enters the phone market faster but is constrained by the old system.

Fifth, Agent OS success depends not only on models, but on task runtime, capability routing, data classification, user confirmation, audit, device-cloud collaboration, and developer relationships.

Sixth, if an OpenAI phone appears around 2028, its greatest impact may come from the sample of a “task-stream-first phone.” Sales may not be the only metric.

For Android phone practitioners, this is not a distant concept. Every system AI entry point, permission dialog, AppFunction, on-device model service, and notification redesign being built now will decide whether the future Agent OS can be caught.

If AI phones are still understood as a collection of gallery, search, summary, and voice-assistant features, this change will be underestimated.

If they are understood as “phones starting to manage tasks,” the question becomes much clearer.

The next phone’s first screen may not immediately lose apps.

But it will look less and less like an app display case.

It will look more like a task entry point that is constrained by the system, authorized by the user, and able to keep working.

References and Sources

• Ming-Chi Kuo’s supply-chain report about OpenAI, MediaTek, Qualcomm, Luxshare, and a 2028 AI Agent phone: Frandroid, Techritual, Qore
• OpenAI and Jony Ive / io hardware background: OpenAI - A letter from Sam & Jony, NPR
• Android AppFunctions documentation: Android Developers - AppFunctions
• A2A protocol materials and Linux Foundation news: A2A Protocol, Linux Foundation A2A project
• Samsung Galaxy AI and system-entry direction: Samsung Newsroom, Samsung MWC 2026
• Doubao phone, Nubia M153, and third-party app restrictions: SCMP, TechNode, Business Standard

This article collects technical questions received after publishing From Trace to Insight: Harness Engineering in SmartPerfetto AI Agent and discusses them in Q&A format.

Q1: Why build a custom YAML Skill system instead of using Claude Code’s standard Skills?

Question context: Claude Code’s Skill system supports placing deterministic scripts in a scripts/ directory to avoid LLM generalization. Since you can use scripts/ to execute fixed SQL, why build a separate YAML Skill system? Isn’t a YAML Skill essentially a tool that lets performance engineers execute SQL according to predefined rules?

Key distinction: The two Skill systems operate at different layers

Claude Code Skills and SmartPerfetto YAML Skills solve problems at different stages:

1
2
3
4
5

Development stage (when I write code):
  Claude Code + Skills/Hooks → helps me develop SmartPerfetto

Runtime stage (when users analyze traces):
  SmartPerfetto Backend + YAML Skills → helps users analyze performance data

Claude Code’s Skills run in the developer’s terminal as CLI tool extensions. SmartPerfetto’s YAML Skills run in the Skill Engine within the Express backend, invoked by the Agent at runtime through the MCP tool invoke_skill. The execution environments, invocation methods, and data flows are completely different.

Even focusing only on “deterministic execution,” YAML Skills have several targeted designs

1. Parameterized SQL, not fixed scripts

Performance analysis SQL isn’t hardcoded – the same Skill needs to accept different parameters (process names, time ranges, frame ID lists):

1
2
3
4
5
6
7
8
9
10

steps:
  - id: thread_state_distribution
    type: atomic
    sql: |
      SELECT state, SUM(dur) as total_dur
      FROM thread_state ts
      JOIN thread_track tt ON ts.track_id = tt.id
      WHERE tt.utid = ${main_thread_utid}
        AND ts.ts BETWEEN ${start_ts} AND ${end_ts}
      GROUP BY state

${main_thread_utid} and ${start_ts} are parameters passed in when Claude calls invoke_skill. The YAML Skill Engine performs parameter substitution before executing the SQL. With scripts/, you’d either write shell scripts that accept parameters and concatenate SQL (prone to injection issues) or write full Python/Node scripts – far more complex than YAML.

2. Self-describing output format (DataEnvelope)

1
2
3
4
5

display:
  level: detail
  columns:
    - { name: state, type: string }
    - { name: total_dur, type: duration }

Each step declares the output columns’ names and types. The frontend automatically renders tables based on this schema – duration types are automatically formatted as ms, timestamp types support click-to-navigate to the Perfetto timeline. With scripts/, the output is free-form text that the frontend can’t automatically render.

3. Composable (composite + iterator)

A composite Skill can reference multiple atomic Skills, and iterators can traverse data rows for per-frame analysis. This composition is declarative in YAML, with the Skill Engine handling orchestration. The scripts/ approach would require writing your own orchestration logic for the same composition.

4. Designed for performance engineers, not developers

The questioner got it right: YAML Skills are essentially a tool that lets performance engineers contribute analysis logic through predefined rules. Performance engineers know which SQL to query and which metrics to examine, but they don’t necessarily know TypeScript. The YAML format lets them directly define SQL queries and output formats without touching backend code. After modifications, changes take effect by simply refreshing the browser in DEV mode.

Comparison summary

The two are not alternatives but solve different problems at different layers.

Q2: How exactly is “deterministic + flexible” implemented?

Question context: The article says “known scenarios use Strategy files to constrain mandatory checks, but within each phase, the specific queries and deep drill directions are autonomously decided by Claude.” Where is the boundary between constraint and autonomy? How exactly is this achieved?

Three-layer mechanism working together

This hybrid design relies on three layers working in concert: Strategy files define “what must be done,” Planning Gate enforces “plan before acting,” and Verifier performs post-hoc checks on “whether it was actually done.”

Layer 1: Strategy files – hard constraints alongside soft guidance

Taking the scrolling analysis scrolling.strategy.md as an example, it defines multiple analysis phases, but the constraint strength differs across phases:

Hard constraints (must execute; skipping triggers verification errors):

Phase 1.9 root cause deep drill is the most strictly constrained phase, with the strategy file using red circle markers and “prohibited” language:

1
2
3
4
5
6
7
8
9
10

**Phase 1.9 -- Root Cause Deep Drill (Red circle mandatory, cannot skip):**

For each reason_code accounting for >15% in `batch_frame_root_cause`,
you **must** select the most severe frame for deep drilling.
**Prohibited:** drawing conclusions solely from `batch_frame_root_cause` statistical classifications.

| Condition                     | Deep drill action                                   |
| Any reason_code Q4>20%        | invoke_skill("blocking_chain_analysis", ...)        |
| binder_overlap >5ms           | invoke_skill("binder_root_cause", ...)              |
| ...

Soft guidance (suggested but skippable):

Phase 1.5 (architecture-aware branching) and Phase 1.7 (root cause branching) use suggestive language like “switch to” and “note,” allowing Claude to decide whether to execute based on actual data:

1
2
3
4
5

**Phase 1.5 -- Architecture-Aware Branching:**

| Architecture | Adjustment action |
| Flutter      | Switch to flutter_scrolling_analysis |
| WebView      | Note CrRendererMain thread |

The entire content of the Strategy file is injected verbatim into the System Prompt, with a hard constraint statement added at injection time:

1
2

Scene Strategy (must be strictly followed)
For the following common scenarios, proven analysis pipelines exist. All phases must be fully executed and cannot be skipped.

Claude sees these phase definitions, red circle markers, and “prohibited” language directly in the System Prompt.

Layer 2: Planning Gate – forces planning first, but doesn’t limit plan content

Before executing any SQL queries or Skill invocations, Claude must first call submit_plan to submit an analysis plan. Calling execute_sql or invoke_skill without submitting a plan is directly rejected:

1
2
3
4

function requirePlan(toolName: string): string | null {
  if (analysisPlanRef.current) return null;  // Plan exists, allow
  return `Must call submit_plan to submit an analysis plan before using ${toolName}`;
}

The key point is: the Gate only requires a plan to exist, not that it precisely matches the Strategy’s phases. Claude can submit any plan structure – it can merge Phase 1 and 1.5, add extra steps not mentioned in the Strategy, or adjust deep drill directions based on preliminary data.

When submitting a plan, the system performs scene-aware keyword checking (e.g., for scrolling scenarios, checking whether the plan mentions “frames,” “jank,” etc.), but this is only at the warning level – the plan is accepted even without these keywords.

The purpose of this design is: force Claude to think clearly about what it wants to do before acting (planning discipline), without restricting how it thinks (planning freedom).

Layer 3: Verifier – multi-dimensional post-hoc checking

There can be gaps between planning and execution – Claude might submit a plan but actually skip a critical step. The Verifier performs multi-dimensional post-hoc checks after analysis completes, primarily using heuristic behavioral checks while supplementing with plan/hypothesis/scene completeness validation:

a) Scene completeness check – whether the analysis output covers the scene’s core content:

1
2
3
4
5
6
7
8
9
10
11

// Scrolling scenario: check if significant jank exists but Phase 1.9 deep drill was skipped
case 'scrolling': {
  const hasSignificantJank = /* detect if text mentions significant frame drops */;
  const hasDeepDrill = /* detect if blocking_chain / binder_root_cause etc. were called */;
  if (hasSignificantJank && !hasDeepDrill) {
    issues.push({
      severity: 'error',
      message: 'Scrolling analysis has jank but missing Phase 1.9 root cause deep drill -- reason_code is just a classification label, not the real root cause'
    });
  }
}

b) Hypothesis closure check – whether all submit_hypothesis calls have corresponding resolve_hypothesis calls.

c) Causal chain depth check – whether CRITICAL/HIGH severity findings contain sufficient causal connectors and mechanistic terminology (heuristic text matching).

d) Optional LLM review – using an independent Haiku model for evidence support verification (can be disabled).

If the check finds ERROR-level issues, it triggers a Correction Prompt for Claude to fill in the gaps.

Note that the Verifier does not check whether Claude’s plan phases match the Strategy’s phase numbers – it checks whether “critical analysis actions are reflected in the output,” not whether “the plan format is correct.”

The complete constraint spectrum

Layering all three mechanisms together, different phases form a spectrum of constraint strength:

Meanwhile, general.strategy.md (the fallback when no scene matches) is entirely soft guidance: it only provides a routing decision tree based on the user’s focus direction (CPU -> cpu_analysis, memory -> memory_analysis), with no mandatory phases. Claude has complete autonomy in the general scenario.

One-sentence summary

Strategy files tell Claude “analyzing scrolling issues requires at least these steps,” Planning Gate ensures it thinks before acting, and Verifier post-checks whether critical steps were actually performed. But within this framework, what specific data to query, which tools to use, and in what order – all of these are autonomously decided by Claude based on actual data.

Q3: What’s the biggest difference between Agent and Workflow? Where are the Agent’s capability boundaries and what determines them?

Question context: In building our Agent, we went from initially assuming the Agent could understand and make decisions about every Skill given to it, to now having essentially hardcoded a decision tree in the Skills. The stepping stones along the way were always: “we assumed the Agent had capability X, but it didn’t,” causing its output to deviate from our expectations, so we kept adding boundaries to the Skills until it became a hardcoded Workflow.

The essential difference: Who holds decision-making authority

Agent and Workflow aren’t two tools or two frameworks – they’re two ends of the same spectrum:

1
2
3
4
5

Hardcoded Workflow <------------------------------------> Fully Autonomous Agent
  |                                                              |
  Developer controls every branch                     LLM decides everything
  |                                                              |
  High determinism, low flexibility                   Low determinism, high flexibility

But in engineering practice, almost no one operates at either extreme of the spectrum. Pure Workflows can’t handle unknown scenarios; pure Agents are unreliable on critical steps. Real-world production systems sit somewhere in the middle.

The root cause of your pitfalls: Making global assumptions about Agent capabilities

“Defaulting to ‘Agent can understand everything’ -> discovering it can’t -> constantly adding constraints -> becoming a hardcoded Workflow” – the fundamental problem with this path is: making a one-size-fits-all judgment about Agent capabilities.

But Agent capabilities vary enormously across different tasks:

The correct approach isn’t “choose Agent globally or choose Workflow globally,” but assign by task:

1
2
3
4
5

Scene recognition     -> Workflow (deterministic logic, no LLM needed)
Data collection       -> Semi-Workflow (Skills define what to query, Agent decides order and params)
Reasoning/attribution -> Agent (this is LLM's core value; given enough data, it performs well)
Output formatting     -> Workflow (templated, deterministic)
Quality verification  -> Workflow (rule checking) + Agent (LLM review)

SmartPerfetto’s approach: A constraint strength spectrum

SmartPerfetto doesn’t choose between Agent and Workflow; instead, it sets different constraint strengths for different analysis phases (detailed in Q2). Here we re-examine this design from the “capability boundary” perspective:

High constraint (Phase 1.9 root cause deep drill) – because the Agent is unreliable at “deciding whether to deep drill”:

1
2
3
4
5
6
7

# Phase 1.9 in scrolling.strategy.md

**Phase 1.9 -- Root Cause Deep Drill (Red circle mandatory, cannot skip):**

For each reason_code accounting for >15% in batch_frame_root_cause,
you **must** select the most severe frame for deep drilling.
**Prohibited:** drawing conclusions solely from batch_frame_root_cause statistical classifications.

Why the hard constraint? Because we found the Agent has a systematic bias: it tends to jump straight to conclusions after getting overview data, skipping the deep drill. This isn’t because the model isn’t smart enough – Claude is perfectly capable of root cause deep drilling – but because the model exhibits “path dependency”: overview data already contains statistical classifications (reason_code), and for the model, “using classification labels directly for conclusions” has much lower cognitive cost than “spending 3 tool-call rounds doing per-frame deep drilling.”

Low constraint (Phase 1.5 architecture branch) – because the Agent is reliable enough at “selecting tools based on data”:

1
2
3
4
5
6
7

# Phase 1.5 in scrolling.strategy.md

**Phase 1.5 -- Architecture-Aware Branching:**

| Architecture | Adjustment action |
| Flutter      | Switch to flutter_scrolling_analysis |
| WebView      | Note CrRendererMain thread |

This uses suggestive language like “switch to” and “note” without enforcement. Because the architecture detection result (Flutter/WebView/Standard) has already been placed in the system prompt by deterministic code, the Agent has a high probability of selecting the correct Skill after seeing this information.

Zero constraint (general scenario) – because the Agent’s autonomous exploration is the only option:

1
2
3
4
5
6
7

# general.strategy.md -- only a routing decision tree, no mandatory steps

Scene: general
priority: 99

The current query did not match a specific scene strategy. Based on the user's
focus direction, use the following decision tree to select the appropriate analysis path.

The general scenario has zero hard constraints because entering general means the user’s question exceeds predefined scenarios, and a Workflow can’t handle it. At this point, the only option is to trust the Agent’s autonomous exploration capability.

What determines Agent capability boundaries

Agent capability boundaries don’t depend on model parameter count or benchmark scores, but on three engineering factors:

1. Observation capability – what data can the Agent “see”

The same model, given structured L2 per-frame data from the scrolling_analysis Skill vs. writing SQL to query raw tables itself, produces significantly different analysis quality. The Agent’s ceiling is determined by the data tools you provide. SmartPerfetto uses 164 YAML Skills to encapsulate domain experts’ query logic; the Agent gets processed, structured analysis data through invoke_skill, not raw millions of trace events.

2. Constraint framework – within what bounds does the Agent make decisions

An unconstrained Agent is like an intern without a task checklist – knowledgeable enough but unsure what to do first. Strategy files, Planning Gate, and Verifier together define the Agent’s decision boundaries: Strategy tells it “at minimum, what must be done,” Planning Gate forces it to “think before acting,” and Verifier post-checks “whether the analysis is sufficient” (heuristic checks + hypothesis closure + scene completeness + optional LLM review).

3. Feedback quality – can the Agent be corrected when wrong

A significant proportion of Agent findings have issues (shallow attribution, false positives, missing critical steps). Relying solely on model self-correction has limited effectiveness. SmartPerfetto uses multi-layer verification + external correction prompts to close the loop:

1
2
3

Verifier finds ERROR -> Generates Correction Prompt -> Triggers SDK retry
                        ^                              |
                  Learned patterns <- Accumulated historical misjudgment patterns

Addendum: Strategy files are SOPs, and that’s fine

Some might point out that scrolling.strategy.md reads like an SOP (Standard Operating Procedure) – with numbered Phases, condition tables, mandatory items, and even explicit invoke_skill("scrolling_analysis", {...}). How is this different from “hardcoding a decision tree in Skills”?

Let’s be direct: in the data collection phase, SmartPerfetto’s scrolling analysis is a Workflow. Strategy files are SOPs that encode domain experts’ analysis experience into deterministic steps. This is intentional.

The key is understanding what the SOP covers and what it doesn’t:

What the SOP can cover (data collection) – what Strategy files do:

1
2
3
4
5

scrolling.strategy.md:
  Phase 1:   "Invoke scrolling_analysis"                <- Hardcoded what data to collect
  Phase 1.5: "Flutter switches to flutter_scrolling_analysis" <- Hardcoded conditional branch
  Phase 1.7: Condition table -> deep drill actions      <- Hardcoded if/then table
  Phase 1.9: "Red circle reason_code >15% must deep drill"  <- Hardcoded mandatory items

What the SOP can’t cover (reasoning/attribution) – where the Agent’s value lies:

1
2
3
4

- Among 47 jank frames, which share the same root cause? (Data clustering)
- Among 19 workload_heavy frames, which is "most severe" and worth deep drilling? (Priority judgment)
- Deep drill reveals Binder blocking 23ms + thermal throttling simultaneously; what's the causal direction? (Causal reasoning)
- How should the final conclusion be organized? How to differentiate recommendations for App developers vs. platform engineers? (Expression decisions)

Different scenarios have different SOP levels:

scrolling.strategy.md has the highest SOP level because scrolling analysis methodology is the most mature. general.strategy.md has almost no SOP because user questions are completely unpredictable.

So the correct understanding is: SmartPerfetto = “SOP-driven data collection + Agent-driven reasoning/attribution.”

The SOP addresses “what data to look at, at minimum, when analyzing scrolling issues” – this question has a deterministic answer, and using an SOP is correct. The Agent addresses “how to reason about causality and organize conclusions after getting the data” – this question differs for every trace and can’t be turned into an SOP.

Back to your pitfalls: The problem isn’t “Skills becoming SOPs” – the data collection phase should use SOPs. The problem is “the SOP consuming the reasoning” – if the SOP hardcodes even the conclusions (“if you see X, output Y”), the Agent truly degrades into a Workflow. The key is letting the SOP stop at data collection and leaving reasoning to the Agent.

One-sentence summary

The difference between Agent and Workflow isn’t “intelligent vs. hardcoded,” but “how decision-making authority is allocated.” Agent capability boundaries are jointly determined by “observation capability x constraint framework x feedback quality.” The correct approach is to allocate decision-making authority by task – granting autonomy where the Agent is reliable, adding constraints where it isn’t – rather than making a one-size-fits-all choice.

Q4: Does the Agent architecture need improvement from a business perspective?

Question context: Agent architecture has gone through different designs and evolutions – from the initial ReAct architecture to LangGraph’s node-based architecture. How do different Agent architecture designs affect it? When building your own business Agent, should you consider the architecture’s impact on Agent performance? For example, SmartPerfetto has added different Skill loading modes on top of the Claude Agent SDK based on business understanding.

Essential differences of three mainstream architectures

Their core difference lies in “who decides what to do next”:

• ReAct: LLM makes complete decisions at every step (what to think, what to do, which tool to use); the framework just forwards
• LangGraph: Developers predefine all possible paths (nodes + edges); LLM only makes local decisions within nodes
• Native SDK: SDK manages the conversation loop, LLM autonomously selects tools, developers indirectly constrain through system prompts and tool design

Why SmartPerfetto chose Native SDK + custom constraint layers

SmartPerfetto’s architecture is Claude Agent SDK (Native SDK) + three constraint layers (Strategy/Planning Gate/Verifier):

1
2
3
4
5
6
7
8

Claude Agent SDK provides:              SmartPerfetto custom-built:
+- Turn loop (auto multi-turn mgmt)     +- Scene Classification (scene routing, <1ms)
+- Tool dispatching (MCP tool calls)     +- Strategy Injection (inject analysis strategy by scene)
+- Streaming (SSE event stream)          +- Planning Gate (force plan before execute)
+- Session resume (multi-turn recovery)  +- Verifier (post-hoc verification + correction retry)
+- Sub-agent orchestration               +- ArtifactStore (3-level cache for token compression)
                                         +- Conditional Tool Loading (inject/hide tools by scene)
                                         +- Cross-Session Memory (pattern memory + negative memory)

Why not LangGraph?

The root cause reasoning paths in performance analysis are unpredictable. The same “scrolling stuttering” could be caused by:

• Binder blocking -> needs to trace thread state on the system_server side
• Slow GPU rendering -> needs to check GPU frequency and fence wait
• GC pauses -> needs to examine Java heap and GC events
• Thermal throttling -> needs to check thermal zone and CPU frequency
• Lock contention -> needs to check monitor contention
• A combination of multiple causes above

With LangGraph, you’d need to predefine a DAG node and jump condition for every root cause path. With 21 reason_codes in performance analysis, each combinable with deep drills, the combinatorial explosion of paths makes the DAG graph unmaintainable.

The more fundamental problem is: before seeing the data, you don’t know which path to take. LangGraph’s DAG graph assumes the developer can predict all branch conditions in advance, but performance analysis branch conditions depend on runtime data.

Advantages of the Native SDK architecture:

LLM autonomously selects tool paths, but three constraint layers ensure critical steps aren’t skipped:

1
2
3
4
5
6
7
8
9

# LangGraph requires predefined DAG:
graph.add_edge("overview", "check_binder")
graph.add_edge("overview", "check_gpu")
graph.add_edge("overview", "check_gc")
# ... every new root cause requires modifying the graph structure

# SmartPerfetto Strategy only needs to declare:
# "For each reason_code accounting for >15%, must select the most severe frame for deep drill"
# Agent autonomously decides which specific deep drill tool to call

Key business-driven architectural designs

The following are architectural improvements SmartPerfetto made based on business understanding, each directly corresponding to a business problem:

1. Conditional tool loading – reduce the Agent’s decision space

SmartPerfetto has up to 20 MCP tools total (9 always-on + 11 conditionally injected), but only a subset is injected per analysis:

1
2
3
4
5
6
7
8
9
10
11

// claudeMcpServer.ts -- switch tool sets by mode

if (options.lightweight) {
  // Factual queries (e.g., "what's the frame rate"): only 3 tools
  toolEntries = [executeSql, invokeSkill, lookupSqlSchema];
} else {
  // Full analysis: 9 always-on (incl. recall_patterns) + conditionally inject by context
  // Comparison mode -> inject compare_skill, execute_sql_on, get_comparison_context
  // Hypothesis management -> inject submit_hypothesis, resolve_hypothesis, flag_uncertainty
  // Planning tools -> inject submit_plan, update_plan_phase, revise_plan
}

Business reason: More tools means higher probability of the Agent selecting the wrong one. A query that just needs a quick answer to “what’s the frame rate,” if presented with a dozen planning/hypothesis/comparison tools, might lead the Agent to over-analyze.

2. Sub-Agent scene gating – avoid unnecessary parallel overhead

1
2
3
4
5
6
7
8
9
10

// claudeAgentDefinitions.ts -- only enable sub-agents in complex scenarios

const ORCHESTRATOR_ONLY_TOOLS = new Set([
  'submit_plan', 'update_plan_phase', 'revise_plan',
  'submit_hypothesis', 'resolve_hypothesis', 'flag_uncertainty',
  'compare_skill', 'execute_sql_on', 'get_comparison_context',
]);

// Sub-agents only get data collection tools, not planning/hypothesis tools
// Design principle: sub-agents collect evidence, orchestrator makes diagnosis

Note: The actual parallelism of sub-agents depends on the SDK’s internal scheduling strategy. We design prompts for parallel evidence collection, but actual execution may be sequential.

3. Lightweight vs Full dual mode – quick Q&A doesn’t go through the full pipeline

When users ask “what’s the frame rate of this trace,” there’s no need to go through the full Planning -> Skill -> Verification pipeline. SmartPerfetto’s ClaudeRuntime performs complexity classification at the entry point:

1
2
3
4
5
6
7
8
9

analyze(query)
  |
queryComplexity === 'quick'
  -> analyzeQuick(): 3 tools, no Planning Gate, no Verifier
  -> Directly answers factual questions

queryComplexity === 'full'
  -> Full pipeline: Planning -> Skill -> Verification -> Correction
  -> Systematic analysis

Business reason: A significant proportion of user questions are factual queries (“what’s the frame rate,” “is there an ANR”), and running the full pipeline would unnecessarily increase latency.

Answering “whether architecture needs business-driven improvement”

Yes, but the improvement direction isn’t switching the underlying framework (ReAct -> LangGraph), but adding business constraint layers on top of the existing framework. Specific recommendations:

1. Extract Strategy files from your Skill decision tree: Rewrite the hardcoded if/else decision logic in code as natural language analysis strategies (Markdown files), injected into the system prompt by scene. This way, domain experts can directly modify analysis logic without touching code.

2. Add a Planning Gate: The requirePlan() implementation is extremely simple (under 10 lines of code), but the effect is significant – forcing the Agent to think before acting empirically reduces going off-track dramatically.

3. Add a post-hoc Verifier: Don’t check whether the Agent’s intermediate steps are “correct” (this is hard to judge); only check whether critical steps “happened” (this is easy to judge).

4. Dynamically adjust tool sets and constraint strength by scenario/complexity: Not all queries need the same analysis depth; give simple queries a fast path.

One-sentence summary

Architecture choice is a business problem, not a technical one. ReAct/LangGraph/SDK are just different implementations of control flow – what truly affects Agent performance is the constraint layer you build on top of the control flow. SmartPerfetto chose Native SDK not because it’s the most advanced, but because performance analysis root cause paths are unpredictable, and predefined DAGs are less effective than letting the Agent explore autonomously within a constraint framework.

Q5: How should a performance AI agent handle scene recognition?

Question context: We want to do scene recognition and route to the correct Skill. When building this, should we rely more on “user utterance” or “logs,” or is there a better approach? Several paths for scene recognition each have issues: code matching (keyword matching leads to results that are too broad or too narrow), LLM understanding (LLM understanding isn’t necessarily accurate), log reconstruction (can filter for scrolling presence, but that may not be what the user cares about).

SmartPerfetto’s approach: Three signal layers with clear division of labor

SmartPerfetto’s scene recognition doesn’t rely on a single signal source but uses three signal layers working together, each solving a different problem:

1
2
3

Layer 1: User utterance -- keyword matching -> scene type (scrolling / startup / anr / ...)
Layer 2: Trace data -- deterministic detection -> architecture info (Flutter / WebView / Standard / Compose)
Layer 3: Data completeness -- table existence checks -> available analysis dimensions (GPU data available? thermal throttling data available?)

Layer 1: User utterance (keyword matching, <1ms)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

// sceneClassifier.ts -- 46 lines of code, handles all scene classification

export function classifyScene(query: string): SceneType {
  const scenes = getRegisteredScenes();  // Loaded from 12 .strategy.md YAML frontmatters
  const lower = query.toLowerCase();

  const sorted = scenes
    .filter(s => s.scene !== 'general')
    .sort((a, b) => a.priority - b.priority);  // ANR(1) > startup(2) > scrolling(3) > ...

  for (const scene of sorted) {
    // Match compound patterns first (more specific, e.g., "startup.*slow")
    if (scene.compoundPatterns.some(p => p.test(query))) return scene.scene;
    // Then match single keywords
    if (scene.keywords.some(k => lower.includes(k))) return scene.scene;
  }
  return 'general';  // Fallback
}

Keywords are defined in each Strategy file’s YAML frontmatter, not hardcoded in TypeScript:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# scrolling.strategy.md frontmatter
keywords:
  - 滑动
  - 卡顿
  - 掉帧
  - jank
  - scroll
  - fps
  - 帧
  - frame
  - 列表
  - 流畅
  - fling
  - stuttering
  - dropped frame
  - 不流畅
  # ... 30+ keywords total

Why keyword matching instead of LLM?

1. Cost: Scene classification executes at the entry of every analysis; keyword matching is <1ms + 0 tokens; LLM calls take ~500ms + ~500 tokens
2. Determinism: The cost of misclassification is very high (injecting the wrong Strategy file); keyword matching behavior is fully predictable
3. Sufficiently accurate: In the performance analysis domain, user queries are highly formatted – someone saying “scrolling stuttering” is asking about scrolling, someone saying “slow startup” is asking about startup. No LLM needed to “understand” this

What about cases where keyword matching falls short?

Keyword matching does have boundaries – when a user says “why is this app slow,” keywords can’t determine whether it’s slow startup or slow scrolling. SmartPerfetto handles this by: falling back to the general scene when no match is found, letting the Agent autonomously choose a direction within the general strategy’s routing decision tree.

1
2
3
4
5
6

# general.strategy.md -- no hard constraints, only routing suggestions

| User focus direction         | Recommended path |
| CPU / scheduling / threads   | invoke_skill("cpu_analysis") |
| Memory / OOM / leaks         | invoke_skill("memory_analysis") |
| Unknown direction            | invoke_skill("scene_reconstruction") -> route by scene |

The core idea of this design is: don’t try to achieve 100% accurate classification at the entry point; instead, let accurate cases take the fast path (keywords -> Strategy), and uncertain cases take the exploration path (general -> Agent autonomous routing).

Layer 2: Trace data – architecture detection (deterministic code)

Scene classification only resolves “what the user wants to analyze,” but the same scenario (e.g., scrolling) requires completely different analysis paths under different rendering architectures:

Architecture detection is delegated to the YAML skill rendering_pipeline_detection – it performs thread/Slice signal collection, pipeline scoring, and sub-variant determination at the SQL layer, supporting 24 fine-grained rendering architectures. The TypeScript side (architectureDetector.ts) is only responsible for calling the skill and mapping results; it doesn’t do direct if/else judgments:

1
2
3
4

rendering_pipeline_detection skill (SQL)
  -> Collect thread signals (1.ui / 1.raster / CrRendererMain / RenderThread ...)
  -> Pipeline scoring (FLUTTER_TEXTUREVIEW / WEBVIEW_BLINK / ANDROID_VIEW_STANDARD ...)
  -> architectureDetector.ts maps to ArchitectureInfo type

Detection results are injected into the system prompt (via templates like arch-flutter.template.md), and the Agent selects the corresponding analysis tools after seeing the architecture information.

Layer 3: Data completeness – capability register

Different trace capture configurations yield different available data dimensions. Some traces lack GPU frequency data; others lack thermal zone data. SmartPerfetto probes the availability of 18 data dimensions before analysis begins:

1
2
3
4
5

frame_rendering: OK (456 rows)
cpu_scheduling: OK (12000 rows)
gpu: MISSING (no gpu_frequency counter)
thermal_throttling: OK (4 zones)
binder_ipc: OK (890 transactions)

This information is likewise injected into the system prompt, telling the Agent which dimensions can be analyzed and which lack data. This prevents the Agent from invoking a Skill with no data backing, getting empty results, then switching directions – such trial-and-error wastes 1-2 tool calls’ worth of tokens.

Evaluating the three paths from the question

1. Code matching (keywords): Viable, but needs well-designed fallback

The questioner said “keyword matching leads to results that are too broad or too narrow.” SmartPerfetto’s experience:

• Priority ordering solves the “too broad” problem: ANR(1) > startup(2) > scrolling(3); when multiple keywords match simultaneously, take the highest priority
• Compound patterns improve precision: /startup.*slow/ is more precise than matching “startup” alone
• general fallback solves the “too narrow” problem: when nothing matches, don’t guess – hand it to Agent for autonomous exploration

2. LLM understanding: Not recommended for the classification entry point; can be used in the fallback path

LLM classification in SmartPerfetto isn’t Layer 1 but rather the Agent’s autonomous routing within the general scenario – at that point, the Agent already has trace data and can make more accurate judgments combined with the data.

3. Log reconstruction: Suitable as a supplementary signal for Layer 2

Logs can tell you “what’s in the trace” (whether scrolling events exist, whether ANR exists), but can’t tell you “what the user cares about.” SmartPerfetto’s data completeness probing plays exactly this role – it doesn’t participate in scene classification but provides the Agent with data availability information.

One-sentence summary

Scene recognition shouldn’t try to solve all problems with a single signal source. Use keyword matching for fast routing (accurate cases), use general fallback for Agent autonomous exploration (uncertain cases), and use trace data for architecture and completeness supplementation. Keyword matching + priority ordering + compound patterns + fallback strategy – 46 lines of code is sufficient.

Q6: How to better leverage AI autonomous exploration in “deterministic steps + AI exploration”?

Question context: We’ve found in production that when AI explores autonomously and drills into root causes, it tends to go off track and give incorrect results. For example, in the SmartPerfetto blog post example – after identifying RenderThread being blocked by Binder in the earlier steps (based on deterministic steps), are the subsequent hypothesis formations and validations pure AI, or do we give the AI some common causes as guidance for Binder blocking and let it investigate on its own?

First, answering the core question: It’s neither pure AI nor hardcoded guidance

SmartPerfetto’s approach is structured reasoning framework + on-demand knowledge injection:

1
2
3
4
5
6
7
8
9
10
11

Deterministic steps produce data
  |
Agent forms hypothesis (autonomous, but constrained by reasoning framework)
  |
Agent selects verification tool (autonomous, but guided by Strategy suggestions)
  |
Verification result feedback
  |
Hypothesis confirmed -> go deeper / Hypothesis rejected -> backtrack and change direction
  |
Verifier post-hoc check

Three key mechanisms make AI autonomous exploration more reliable:

Mechanism 1: Hypothesis management tools – adding structure to the reasoning process

SmartPerfetto provides submit_hypothesis and resolve_hypothesis as two MCP tools. Instead of letting the Agent reason implicitly in internal monologue, it forces externalization:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Agent calls:
  submit_hypothesis({
    description: "system_server slow Binder response causing RenderThread blocking",
    expected_evidence: "system_server's corresponding Binder transaction thread_state shows prolonged Runnable/Sleeping"
  })

Agent calls:
  execute_sql("SELECT ... FROM thread_state WHERE utid = ... AND ts BETWEEN ...")

Agent calls:
  resolve_hypothesis({
    id: "h1",
    outcome: "rejected",
    evidence: "system_server Binder thread state is normal, response time <2ms,
              RenderThread blocking cause is dequeueBuffer waiting for SurfaceFlinger"
  })

Why is this effective? The hypothesis management tools force the Agent to explicitly declare “what I’m verifying” and “what I expect to see” before taking action. This has two benefits:

1. Prevents goal drift – the Agent won’t forget what it originally wanted to verify while collecting data
2. Auditable – every hypothesis has a complete record; the Verifier can check whether all hypotheses were resolved

Mechanism 2: Knowledge injection – not hardcoded guidance, but on-demand domain knowledge loading

“Could we give the AI some common causes as guidance for Binder blocking?”

Yes, but not hardcoded in Skills. Instead, it’s loaded on demand through the lookup_knowledge MCP tool. After discovering Binder blocking, the Agent can call:

1

invoke lookup_knowledge("binder-ipc")

This returns a Binder IPC knowledge template (knowledge-binder-ipc.template.md) containing:

• Classification of typical Binder transaction blocking causes (server-side busy, process frozen, CPU scheduling delay, oneway queue full)
• Investigation paths and key metrics for each cause
• Common misdiagnosis scenarios (e.g., oneway transactions don’t block the caller)

Key design: Knowledge is actively pulled by the Agent, not force-injected by the system. There are currently 8 knowledge templates (rendering-pipeline, binder-ipc, gc-dynamics, cpu-scheduler, thermal-throttling, lock-contention, startup-root-causes, data-sources). If all templates were pre-injected into the system prompt, it would consume a large number of tokens and most would be irrelevant. Through on-demand loading via MCP tools, the Agent only retrieves the relevant domain background knowledge when needed.

SmartPerfetto also provides conditional deep drill suggestion tables in Strategy files, which is another form of guidance:

1
2
3
4
5
6
7
8

# scrolling.strategy.md Phase 1.9

| Condition                         | Deep drill action |
| Any reason_code Q4>20%            | invoke_skill("blocking_chain_analysis", ...) |
| binder_overlap >5ms               | invoke_skill("binder_root_cause", ...) |
| cpu_runnable_ratio >30%           | invoke_skill("cpu_analysis", ...) |
| thermal_throttle detected         | invoke_skill("thermal_throttling", ...) |
| gc_pause_total >10ms              | invoke_skill("gc_analysis", ...) |

This table isn’t a hardcoded decision tree – it’s a lookup table for the Agent. The Agent decides which row to follow based on metric values in the data. If the data doesn’t match any row, the Agent can explore autonomously.

Mechanism 3: ReAct Reasoning Nudge – triggering reflection when tools return results

During the first few successful returns from data tools (execute_sql / invoke_skill), SmartPerfetto appends a reasoning prompt at the end of the result:

1
2
3
4
5
6
7
8

// claudeMcpServer.ts

const REASONING_NUDGE = '\n\n[REFLECT] Before executing the next step: ' +
  'What are the key findings from this data? Does it support/refute your hypothesis? ' +
  'If there are important inferences, please record them using submit_hypothesis or write_analysis_note.';

// Only appended during the first N data tool calls, then stopped (to control token cost)
const REASONING_NUDGE_MAX_CALLS = 4;

Extremely low cost (~20 tokens/call, ~80 tokens total for the first 4 calls), but significantly effective. The reason it’s not applied throughout is to control token overhead in the latter half of analysis – the first few nudges already establish the pattern of “receive data -> reflect first -> then act.” Without this nudge, the Agent tends to call tools consecutively without pausing to think – collecting data 5 times without forming any intermediate conclusions, resulting in poor final summary quality.

Walking through the complete flow with the Binder example from the article

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

1. Start with overview -> discovers 47 jank frames, P90 = 23.5ms
   [Deterministic: invoke_skill("scrolling_analysis")]

2. Decide direction based on overview -> 40% stuck in APP phase, prioritize App side
   [Deterministic: data-driven, Strategy file suggestion]

3. Select representative frame for deep drill -> Frame #234 RenderThread blocked by Binder for 23ms
   [Deterministic: invoke_skill("jank_frame_detail")]

--- AI autonomous exploration begins below ---

4. Agent proactively loads knowledge: lookup_knowledge("binder-ipc")
   -> Gets Binder blocking common cause classification table
   [AI decision + knowledge guidance]

5. Agent forms hypothesis: submit_hypothesis("system_server slow Binder response")
   Expected evidence: server-side thread_state showing prolonged Sleeping/Runnable
   [AI reasoning, hypothesis tool forces externalization]

6. Agent verifies: execute_sql("query Binder counterpart thread_state")
   -> Discovers system_server CPU scheduling delay, not slow Binder response
   [AI autonomous tool call]

7. [REFLECT] nudge triggers reflection
   Agent: "Hypothesis 1 doesn't hold; server-side thread_state shows Runnable queuing,
          actual cause is CPU scheduling delay"
   -> resolve_hypothesis(outcome: "refined",
       evidence: "system_server CPU scheduling delay causing Binder thread queuing")
   [AI reasoning + investigation path from knowledge template]

8. Agent goes deeper: execute_sql("query CPU frequency + thermal zone")
   -> Discovers thermal throttling, CPU big cores frequency-limited to 50%
   [AI autonomously selects next deep drill direction]

9. Comprehensive conclusion: RenderThread Binder blocking <- system_server CPU scheduling delay <- thermal throttling
   [AI synthesis, forming WHY chain]

--- Verifier post-hoc check ---

10. Verifier heuristic checks:
    - Text pattern matching: does the conclusion reflect deep drill analysis (not just referencing overview data)
    - Hypothesis closure: do all submit_hypothesis calls have corresponding resolve_hypothesis calls
    - Scene completeness: does scrolling analysis include frame/jank related content
    - Causal chain heuristic: are there sufficient causal connectors and mechanistic terminology
    [Deterministic: heuristic rule checks, not precise verification]

Note that Steps 4-9 are all AI autonomous exploration, but constrained by three mechanisms:

• Hypothesis tools force reasoning externalization (Steps 5, 7)
• Knowledge injection provides domain investigation paths (Step 4)
• REFLECT nudge triggers reflection after the first few tool returns (Step 7)

Four practical recommendations for making AI autonomous exploration more reliable

1. Give data, not conclusions

Skills should return structured data (frame durations, thread state distributions, blocking function lists), not pre-drawn conclusions (“RenderThread blocking is caused by Binder”). Having AI reason its own conclusions from data is more reliable than having it build further analysis on conclusions provided by others.

2. Give framework, not path

Strategy files should define “what must be done” (Phase 1.9 must deep drill), not “how to do it” (first query A, then B, then C). An Agent autonomously selecting paths within a framework constraint is far more reliable than free exploration without any constraints, yet far more flexible than hardcoded paths.

3. Give knowledge, not answers

Knowledge templates should contain “possible cause classifications and investigation methods,” not “if you see X, it’s Y.” The former helps the Agent build a reasoning framework; the latter turns the Agent back into a Workflow.

4. Verify behavior, not conclusions

Verifiers should use heuristic rules to check “whether the analysis output reflects critical actions” (does the conclusion show traces of deep drill analysis, are all hypotheses resolved, does the causal chain have sufficient depth), rather than trying to judge “whether the conclusion is correct” (this should be done offline with LLM Judge evaluation, not at runtime). Note that these are text pattern matching level heuristic checks, not precise tool call log audits.

One-sentence summary

AI autonomous exploration reliability isn’t guaranteed by “hardcoded guidance,” but by three mechanisms: hypothesis management tools externalize reasoning, on-demand knowledge injection provides domain investigation frameworks, and ReAct nudge prevents blind tool calling. The key principles are “give data not conclusions, give framework not path, give knowledge not answers.”

Q7: How is the Prompt assembled for each turn?

Question context: LLM Agent output quality depends heavily on system prompt design. How does SmartPerfetto construct the prompt for each analysis? How does the prompt change across different scenarios and different turns? How is the token budget controlled?

Overall design: Four-tier layered assembly + cache optimization

SmartPerfetto’s system prompt isn’t a static string but is dynamically assembled by the buildSystemPrompt() function (claudeSystemPrompt.ts:260) before each SDK query. The assembly follows a core principle:

Sort by “stability” – content that changes less frequently goes first, more dynamic content goes last.

The reason for this design is Anthropic API’s automatic caching mechanism: when the system prompt exceeds 1024 tokens, the API automatically caches the prompt prefix. By placing unchanging content at the very front, most of the prompt can hit the cache across multi-turn conversations, significantly reducing latency and cost:

1
2
3

Same trace + same scene:     ~4000 tokens cached (~80% savings)
Same trace + different scene: ~800 tokens cached  (~18% savings)
Different trace:              ~400 tokens cached   (~8% savings)

Four-tier assembly structure

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

+-------------------------------------------------------+
|  Tier 1: STATIC (unchanged within process lifetime)    |
|  +---------------------------------------------------+ |
|  | prompt-role.template.md        (~200 tokens)       | |
|  | -> Role definition: Android performance expert     | |
|  +---------------------------------------------------+ |
|  | prompt-output-format.template.md (~850 tokens)     | |
|  | -> Output format: [CRITICAL]/[HIGH]/[MEDIUM]/[LOW] | |
|  | -> Root cause reasoning chain format, Mermaid rules| |
|  | -> Slice nesting rules, CPU frequency estimation   | |
|  +---------------------------------------------------+ |
+-------------------------------------------------------+
|  Tier 2: PER-TRACE (stable within same trace)          |
|  +---------------------------------------------------+ |
|  | Architecture info          (~150 tokens)            | |
|  | -> "Flutter TextureView, confidence 92%"            | |
|  | -> + arch-flutter.template.md architecture guide    | |
|  +---------------------------------------------------+ |
|  | Focus application          (~100 tokens)            | |
|  | -> "com.example.app (primary focus), 456 frames"    | |
|  +---------------------------------------------------+ |
|  | Data completeness          (~200 tokens, droppable) | |
|  | -> Only reports missing/insufficient dimensions     | |
|  +---------------------------------------------------+ |
|  | SQL knowledge base ref     (~300 tokens, droppable) | |
|  | -> Tables/views/functions matched from stdlib index | |
|  +---------------------------------------------------+ |
+-------------------------------------------------------+
|  Tier 3: PER-QUERY (changes with scene/query)          |
|  +---------------------------------------------------+ |
|  | Methodology + scene strategy (~1200 tokens)         | |
|  | -> prompt-methodology.template.md                   | |
|  | -> {{sceneStrategy}} = scrolling.strategy.md        | |
|  |   or one of 12 strategies (startup/anr/general...) | |
|  +---------------------------------------------------+ |
|  | Sub-agent collaboration     (~200 tokens, droppable)| |
|  | -> When to delegate vs direct call                  | |
|  | -> Scrolling-specific parallel evidence collection  | |
|  +---------------------------------------------------+ |
+-------------------------------------------------------+
|  Tier 4: PER-INTERACTION (may change every query)      |
|  +---------------------------------------------------+ |
|  | User selection context     (~300 tokens, not drop.) | |
|  | -> Time range selection (selection-area.template.md)| |
|  | -> Or Slice selection (selection-slice.template.md) | |
|  +---------------------------------------------------+ |
|  | Comparison mode context    (conditionally injected) | |
|  | -> Dual-trace comparison methodology + tool guide   | |
|  +---------------------------------------------------+ |
|  | Conversation context       (~500 tokens)            | |
|  | -> Analysis notes (<=10, sorted by priority)        | |
|  | -> Previous findings (<=10)                         | |
|  | -> Known entities (for drill-down)                  | |
|  | -> Conversation summary (cross-turn compressed,     | |
|  |    <=2000 tokens)                                   | |
|  +---------------------------------------------------+ |
|  | SQL pitfall records        (<=5, droppable)         | |
|  | -> ERROR -> BAD SQL -> FIX SQL                      | |
|  +---------------------------------------------------+ |
|  | Historical analysis patterns (cross-session, drop.) | |
|  | Historical pitfall records  (cross-session, drop.)  | |
|  +---------------------------------------------------+ |
|  | Historical analysis plans  (<=3 turns, droppable)   | |
|  | -> Phase status: done/skipped/pending + summary     | |
|  +---------------------------------------------------+ |
+-------------------------------------------------------+

Template loading and variable substitution

All prompt content is defined in Markdown files (detailed in Q1); TypeScript only handles loading and variable substitution:

1
2
3
4
5
6
7
8
9
10
11

// strategyLoader.ts -- template system

// 1. Load template (DEV mode skips cache, browser refresh takes effect)
loadPromptTemplate('prompt-methodology')  // -> strategies/prompt-methodology.template.md

// 2. Load scene strategy (extract keywords from YAML frontmatter, body as strategy content)
getStrategyContent('scrolling')  // -> Markdown body of strategies/scrolling.strategy.md

// 3. Variable substitution
renderTemplate(methodologyTemplate, { sceneStrategy })
// Replaces {{sceneStrategy}} with scrolling.strategy.md content

Template file inventory:

Token budget management

Budget ceiling: 4500 tokens (MAX_PROMPT_TOKENS). During correction retries, if SDK auto-compact is detected (conversation history automatically compressed), the budget is reduced to 3000 tokens to leave room; otherwise the original prompt is reused.

Token estimation method: Mixed Chinese-English estimation – Chinese characters at 1.5 tokens/character, ASCII at 0.3 tokens/character. This is a rough approximation but sufficiently accurate for budget management.

Progressive dropping strategy when over budget:

When the assembled prompt’s token count exceeds the budget, entire sections are dropped in order from lowest to highest priority:

1
2
3
4
5
6
7
8

Drop order (dropped first -> dropped last):
1. Perfetto SQL knowledge base reference -> Agent can use lookup_sql_schema tool instead
2. Trace data completeness              -> Helpful but Agent can discover missing data at runtime
3. Historical analysis patterns          -> Cross-session pattern memory, non-critical
4. Historical pitfall records            -> Cross-session negative memory, droppable
5. SQL pitfall records                   -> Nice to have
6. Sub-agent collaboration               -> Only useful when sub-agents are enabled
7. Historical analysis plans             -> Supplementary context

Content that is never dropped:

• Role definition, output format (Tier 1 static)
• Architecture info, focus application (Tier 2 per-trace)
• Methodology + scene strategy (Tier 3 core)
• User selection context (user’s explicit intent)
• Conversation context (previous findings and analysis notes)

Complete context construction flow

In ClaudeRuntime.analyze(), prompt assembly is preceded by over twenty preparation phases to collect all context:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

Phase 0:   Selection context logging
Phase 0.5: Focus app detection (3 methods: battery_stats / oom_adj / frame_timeline)
Phase 1:   Skill executor initialization
Phase 2:   Architecture detection (LRU cached, detected once per trace)
Phase 2.5: Vendor detection (OEM customization, LRU cached)
Phase 2.8: Comparison mode context (dual-trace mode)
Phase 2.9: Data completeness probing (18 dimensions, ~50ms)
Phase 3:   Session context + conversation history
Phase 4:   Entity store (drill-down references)
Phase 5:   Scene classification (keyword matching, <1ms)
Phase 5.5: Cross-session pattern memory matching
Phase 6:   ArtifactStore + analysis notes
Phase 6.5: Analysis plan (current + historical)
Phase 6.6: Watchdog feedback references
Phase 6.7: Hypothesis state
Phase 6.8: Uncertainty flags
Phase 7:   SQL error tracking
Phase 8:   MCP Server creation (inject all the above state)
Phase 9:   (removed)
Phase 10:  SQL knowledge base context
Phase 11:  Sub-agent definitions
Phase 12:  SQL error-fix pairs
Phase 13:  -> buildSystemPrompt(context) -> final prompt

All Phase results feed into the ClaudeAnalysisContext object, passed to buildSystemPrompt() for final assembly.

Quick vs Full dual mode

Not all queries need the full 4500-token prompt. When users ask factual questions (e.g., “what’s the frame rate”), SmartPerfetto uses a streamlined quick prompt:

1
2
3
4

// buildQuickSystemPrompt() -- ~1500 tokens
// Loads prompt-quick.template.md
// Only injects {{architectureContext}} and {{focusAppContext}}
// No methodology, no scene strategy, no conversation context

How the prompt changes across multi-turn conversations

In multi-turn analysis (user follow-ups or drill-downs), prompt changes depend on whether the SDK session hits resume:

Turn 1: No conversation context, no historical plans, no analysis notes

Turn 2 onward – SDK session resume hit (within 4 hours):

• SDK internally already holds complete conversation history; previousFindings and conversationSummary are not re-injected
• But still injected: analysis notes (<=10), entity context (drill-down references), historical plans (<=3 turns)
• Tiers 1-3 remain unchanged, hitting ~80% cache

Turn 2 onward – SDK session expired or unavailable:

• Previous turn’s findings are manually injected as “previous analysis findings” (<=10)
• Conversation summary is manually injected (sessionContext.generatePromptContext(2000), <=2000 tokens)
• Analysis notes, entity context, and historical plans same as above

Correction retry turn: If SDK auto-compact is detected (conversation history automatically compressed), token budget drops from 4500 to 3000, and progressive dropping more aggressively removes non-critical sections. If auto-compact hasn’t occurred, the original system prompt is reused.

A concrete example: Prompt assembly for scrolling analysis

User inputs "Analyze scrolling stuttering", Flutter TextureView architecture, Turn 1:

1
2
3
4
5
6
7
8
9
10

[Tier 1] prompt-role.template.md                    -> "You are an Android performance analysis expert..."
[Tier 1] prompt-output-format.template.md            -> Output format rules
[Tier 2] "Architecture: Flutter TextureView, confidence 92%"  -> + arch-flutter.template.md
[Tier 2] "Focus app: com.example.app (primary focus)"  -> frame count + detection method
[Tier 2] "Data completeness: gpu MISSING, suspected not captured"  -> only reports missing/insufficient
[Tier 2] SQL knowledge base: android_frames, slice_self_dur  -> stdlib match results
[Tier 3] prompt-methodology + scrolling.strategy.md   -> Phase 1->1.5->1.9->3 complete strategy
[Tier 4] (no selection, no conversation context, no historical plans)

Estimated: ~3200 tokens, within 4500 budget, no dropping needed

User follows up with "Deep dive into frame 3", Turn 2 (SDK session resume hit):

1
2
3
4
5
6
7
8

[Tier 1-3] Same as Turn 1 (hitting ~80% cache)
[Tier 4] Conversation context (SDK holds conversation history, findings not re-injected):
  - Analysis notes: "Warning [Hypothesis] Primary jank cause is RenderThread Binder blocking"
  - Entity context: frame#3's ID and time range
  - Historical plan: "Done Phase 1 overview, Done Phase 1.9 deep drill, Pending Phase 3 conclusion"
  (previousFindings and conversationSummary managed internally by SDK session, not duplicated in prompt)

Estimated: ~3400 tokens, within budget

One-sentence summary

The prompt is sorted by “stability” across four tiers (Static -> Per-Trace -> Per-Query -> Dynamic), leveraging API prefix caching to achieve ~80% token savings across multi-turn conversations. The template system lets domain experts directly edit analysis strategies without touching TypeScript. When over budget, progressive dropping by priority occurs, but role definition, scene strategy, and user selection are always preserved – these three determine the analysis direction and scope.

Q8: What Skills does SmartPerfetto have?

Question context: SmartPerfetto’s analysis capabilities are carried by YAML Skills. A complete Skill inventory helps understand the system’s analysis coverage.

Overview

Atomic Skills (87)

Single-step data extraction and detection – the building blocks for all higher-level Skills.

Frame rendering and jank:

VSync and refresh rate:

CPU and scheduling:

GPU:

Main thread analysis:

Binder IPC:

Locks and synchronization:

Startup-specific (19):

Memory and GC:

Input and touch:

System and device:

Others:

Composite Skills (29)

Combine multiple atomic skills, supporting iterator (per-frame/per-event deep drill) and conditional (data-driven branching).

Deep Skills (2)

Deep profiling, typically requiring longer execution time.

Pipeline Skills (28)

Rendering pipeline detection + teaching. Each pipeline skill corresponds to a rendering architecture, including pipeline description, key threads, performance metrics, and optimization recommendations.

Note: _base.skill.yaml is the base template file for Pipeline Skills, not registered as an available Skill, and not counted in the total.

Module Skills (18)

Modular analysis configuration, organized by layer. The Agent discovers them via list_skills and invokes on demand.

Hardware layer (5):

Framework layer (6):

Kernel layer (4):

Application layer (3):

Relationships between Skills

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Module Skills (configuration layer)
  +-> Define analysis scope and focus areas

Composite Skills (orchestration layer)
  +-> Reference multiple Atomic Skills
  +-> Iterator: per-frame/per-event traversal deep drill
  +-> Conditional: data-driven branching

Atomic Skills (execution layer)
  +-> Directly execute SQL, return DataEnvelope

Pipeline Skills (knowledge layer)
  +-> Rendering pipeline teaching + detection

Deep Skills (profiling layer)
  +-> Callstack / CPU profiling deep analysis

Agent’s typical invocation path (scrolling analysis example):

1
2
3
4
5
6

invoke_skill("scrolling_analysis")          <- Composite, internally calls multiple Atomic
  -> consumer_jank_detection                 <- Atomic, detect frame drops
  -> per-frame iterator -> jank_frame_detail <- Composite, deep drill per frame
    -> main_thread_states_in_range           <- Atomic
    -> binder_blocking_in_range              <- Atomic
    -> frame_blocking_calls                  <- Atomic

(Continuously updated; new questions will be added as received)

This article documents the key technical decisions made while building SmartPerfetto from scratch – why certain approaches were chosen over others, where pitfalls were encountered, and how the design was adjusted afterward.

Why Build This Tool

I have spent years working on Android performance optimization. A significant portion of my daily work involves analyzing Perfetto traces – Perfetto is Google’s open-source system-level tracing tool that captures frame rendering, thread scheduling, CPU frequency, Binder IPC, and other data, making it essentially the standard tool for Android performance analysis. Its trace_processor engine loads traces into an embedded SQLite database, enabling SQL-based queries.

The trace analysis process is highly repetitive: locate the problem interval, inspect frame data, examine thread states, trace the blocking chain, and correlate system metrics. The steps are similar each time, but the details differ for every trace. This pattern of “fixed workflow, varying details” is well-suited for an AI Agent – automate the data collection and initial attribution within the fixed workflow, while humans make the final judgments and confirmations.

SmartPerfetto is the product of this experiment. It adds an AI analysis panel on top of the Perfetto UI, where users ask questions in natural language (e.g., “analyze scrolling performance”), and a Claude Agent uses MCP (Model Context Protocol, Anthropic’s tool-calling protocol) behind the scenes to execute SQL queries via trace_processor, autonomously completing multiple rounds of data collection and analysis.

The purpose of this article is to document the engineering decisions and lessons learned during the construction process. From the initial “just call the API directly” approach to the current system with up to 20 MCP tools (9 always-on + 11 conditionally injected) + 164 YAML Skills + a three-layer verification system, every design choice was driven by concrete counterexamples – approaches that were tried and failed before being replaced. These hard-won lessons can be directly applied by engineers building AI Agent applications or Android performance tools.

Opening: Two Analysis Paths for the Same Trace

A scrolling trace from a 120Hz device, where the user reports occasional stuttering during list scrolling. Opening it in Perfetto reveals 18 dropped frames during the inertial scrolling phase, with 3 Full-severity frames (~60ms, where the per-frame budget on a 120Hz device is 8.33ms).

Jank (Dropped Frames): Perfetto’s frame_timeline track records the rendering duration of each frame. Exceeding the VSync period (8.33ms at 120Hz) results in user-perceptible stuttering. The jank_type field distinguishes jank types: App-side timeouts, SurfaceFlinger composition delays, Buffer Stuffing (BufferQueue backpressure), etc.

Path A: Manual Analysis

1
2
3
4
5
6
7
8
9
10

1. Open Perfetto UI, drag the timeline to locate the scrolling interval
2. Expand the frame_timeline track, inspect each frame that exceeds the VSync period
3. 18 dropped frames -- expand each one, examine thread_state slices to see what the main thread was doing
4. Frame 1: Sleeping, manually check waker_utid -> system_server (the core Android system process hosting AMS/WMS and other system services) Binder response was slow
   Frame 2: Running, but stuck in Choreographer#doFrame -> RecyclerView onBind is too heavy
   Frame 3: Sleeping + Running alternating -> dequeueBuffer waiting for SurfaceFlinger composition
   ... (15 more frames to check one by one)
5. Correlate with CPU frequency tracks to check for thermal throttling or governor frequency ramp-up delays
6. Check for GC pauses, Lock contention, Binder timeouts
7. Aggregate evidence and organize conclusions

thread_state records a thread’s scheduling state (Running / Runnable / Sleeping / Uninterruptible Sleep, etc.). Different states point to different investigation directions – Runnable typically indicates CPU scheduling issues, while Sleeping typically indicates waiting/blocking issues. The waker_utid field records the source thread ID that woke up the thread, helping trace cross-process blocking chains.

Steps 3-4 represent the bulk of the work – 18 dropped frames, each requiring expanding thread_state, tracing the blocking cause, and correlating CPU scheduling. The analysis process is serial, frame by frame: the drill-down path may differ for each frame (Binder? Lock? GC? IO?), and all must be examined before drawing conclusions.

Path B: SmartPerfetto Agent

The user inputs "analyze scrolling performance", and the following shows the Agent’s actual operations (from session log session_agent-1774679540422):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

classifyScene("analyze scrolling performance") -> scrolling  (<1ms, keyword matching)
buildSystemPrompt() -> 4500 tokens (scrolling.strategy.md injected)

submit_plan({
  phases: ["p0: architecture + time range", "p1: scrolling overview + jank list",
           "p1_9: root cause deep drill", "p3: comprehensive conclusion"],
  successCriteria: "WHY chain >= 2 levels, actionable optimization suggestions"
})

invoke_skill("scrolling_analysis", {process_name: "com.example..."})
  -> All 18 dropped frames extracted, each with:
    - Duration, jank_type, reason_code (App/SF/GPU)
    - Main thread state distribution (Running/Sleeping/IO)
    - Top-3 blocking functions
  -> ArtifactStore: art_1 (~440 tokens reference)

fetch_artifact("art_1", "rows", 0, 18)  -> Retrieve structured data for all 18 frames
invoke_skill("frame_blocking_calls", {frame_ids: [3, 7, 12]})
  -> Blocking chains for 3 representative frames: dequeueBuffer / Binder / GC
lookup_knowledge("rendering-pipeline")  -> Load rendering pipeline reference material

Output conclusion:
  [HIGH] 18 jank frames during inertial scrolling, 3 Full-severity ~60ms drops
  [MEDIUM] 14 Buffer Stuffing frames (App finished rendering on time, but BufferQueue was full causing presentation delay)
  [MEDIUM] Vulkan Shader first-frame compilation + CPU cold frequency (18.66ms, 2.2x over budget)

Metrics snapshot (from logs/metrics/): 16 tool calls, 0 failures, SQL average 652ms.

The following diagram shows the request lifecycle of a complete analysis – every step from user input to final conclusion:

Both paths follow the same analysis steps – query frame data -> locate jank -> trace blocking chains -> correlate system state -> synthesize conclusions.

The difference is: manual analysis processes frames serially, with each frame requiring manual expansion and tracing; the Agent uses the scrolling_analysis Skill to batch-retrieve structured data for all 18 frames via a single SQL query, then selects representative frames for deep-drill blocking chain analysis.

The Agent’s analysis results are also rendered directly in the Perfetto UI:

• Auto-Pin: Key frames and slices mentioned by the Agent are automatically marked on the timeline
• Click-to-Navigate: Timestamps and frame IDs in conclusions support click-to-navigate to the corresponding position in Perfetto
• Data Tables: Complete performance data for all 18 frames is rendered as sortable, filterable structured tables

Screenshot: SmartPerfetto frontend exists as a Perfetto plugin

Screenshot: During scrolling analysis, each dropped frame is analyzed in detail – click the arrow on the left to expand

Screenshot: Scrolling analysis conclusion

Screenshot: Scrolling analysis conclusion, representative frame analysis

Screenshot: Scrolling analysis conclusion, representative frame analysis

Screenshot: Each analysis round has its own analysis report, with content consistent with (and more detailed than) the frontend display

Analysis conclusions, data tables, and the Perfetto timeline are all on the same interface. After the Agent completes batch data collection and initial attribution, engineers confirm key findings directly in the Perfetto UI.

It should be noted that the current Agent still requires human judgment in complex edge cases (misdiagnosis issues are discussed in detail later). This article documents the engineering decision-making process behind building this Agent.

Part 1: Why Can’t an LLM Directly Analyze a Trace?

Before discussing the architecture, a fundamental question must be answered first: why can’t we simply send trace data to an LLM and let it analyze? The answer to this question determines the starting point of SmartPerfetto’s entire architecture.

Data Scale: Traces Don’t Fit in Context

The data scale of a real Perfetto trace looks like this:

The gap is several orders of magnitude. Even a relatively small 50MB trace, with its slices (function call records), counters (CPU frequency sample points), thread_state (thread scheduling states) and other data, far exceeds the LLM’s context capacity when serialized.

This means the LLM cannot possibly “see” the trace data directly. It must query on demand through tools – first use SQL to find the needed data subset (e.g., the state distribution of a specific thread within a certain time range), then analyze the query results. This constraint fundamentally dictates that SmartPerfetto must be a tool-driven Agent architecture, rather than a simple approach of feeding data into the prompt.

Precise Computation: LLMs Are Poor at Numerical Processing

The daily work of performance analysis revolves around precise numerical values: P50/P90/P99 percentiles of frame durations, VSync period detection (requiring median calculation of VSYNC-sf intervals and snapping to standard refresh rates), CPU utilization percentages, and time proportions of various thread states.

LLMs frequently make errors with these numerical computations. A real example: during early testing, Claude judged a 16.7ms frame duration as “normal, not exceeding VSync period” – it was using the 60Hz (16.67ms) frame budget. But this trace was captured from a 120Hz device, where the per-frame budget should be 8.33ms, making 16.7ms actually twice over budget. Such errors may seem small, but in performance analysis they lead to completely opposite conclusions.

Numerical computation must be handled by tools – SQL’s AVG(), PERCENTILE() and the predefined statistical logic in YAML Skills ensure consistent and precise results every time.

Domain Knowledge: LLMs Know It but Can’t Apply It

The complexity of Android’s rendering pipeline exceeds many developers’ expectations. The three most common rendering paths are: the standard HWUI pipeline (HWUI is Android’s default hardware-accelerated rendering engine, where View drawing instructions are generated on the main thread, submitted to the GPU by RenderThread, and finally composited to the screen by SurfaceFlinger), Flutter’s dual-thread model (1.ui -> 1.raster, bypassing RenderThread), and WebView’s Chromium pipeline (CrRendererMain thread handles rendering). Beyond these, there are also Jetpack Compose, game engines, camera pipelines, and more. SmartPerfetto’s architecture detection system currently identifies 24+ rendering pipelines, and jank analysis for different pipelines requires examining different threads and metrics – which is why architecture detection is the first step of analysis.

The root cause of stuttering may span threads (main thread blocked -> cause is in RenderThread), span processes (App waiting -> system_server’s WindowManagerService responds slowly), or even span hardware layers (CPU scheduled to a little core -> insufficient compute power -> frame timeout).

The LLM’s training data includes these concepts – it “knows” what RenderThread is, what Binder is, what SurfaceFlinger is. But when facing a specific trace, it lacks the ability to apply this knowledge in a phased, scene-specific manner. For example, when analyzing scrolling jank, one needs to first check frame-level data (which frames dropped, what type of jank), then choose different deep-drill paths based on the predominant root cause type (App-side blocking goes to blocking_chain_analysis, compositor-side delays go to SurfaceFlinger analysis). This kind of step-by-step, conditional-branching analysis workflow requires strategy injection for guidance.

Reliability: Error Rates Are High in Practice

Even with the data access problem solved, having an LLM directly produce performance analysis conclusions still faces reliability issues. In SmartPerfetto’s actual operation, I observed several typical output problems:

• Hallucinations: Generating data or metrics that don’t exist in the trace
• Omissions: Missing critical checks (e.g., not checking the impact of JIT compilation and class loading when analyzing startup performance)
• Shallow attribution: Stopping at “main thread busy” without further tracing whether it’s busy with futex (lock contention), binder_wait (cross-process waiting), or GC pause
• Inconsistent conclusions: Analyzing the same trace twice and getting different severity assessments

Part 2 later in the article discusses this issue in detail – a quality audit 18 days after agentv3 went live showed that approximately 30% of Agent conclusions contained varying degrees of misjudgment.

SmartPerfetto’s Division of Labor

Based on these four problems, SmartPerfetto’s architecture divides responsibilities as follows: