RFC-0017: Tracing with Out-of-Tree Protos

[github-actions[bot]]

📄 RFC Doc: 0017-out-of-tree-protos.md

---

Tracing with Out-of-Tree Protos

Authors: @primiano
Status: Draft
PR: N/A
Internal bug: b/483721243

Problem

Going forward we are expecting more and more contributions to trace points
from code all over Android platform. Today when an Android developer wants to
define a custom TrackEvent proto (which is something that we highly encourage
to prevent stringification of args) they have to edit
android_track_event.proto which involves a
GitHub PR.

This is highly inconvenient as it involves cross-repo changes and waiting for
autorollers.

Note that this problem is not just Android-specific and generally applies to
any other user of perfetto. This RFC focuses on Android to keep the discussion
concrete, but the solution generalizes.

Note also that this problem extends also to the lower-level DataSource SDK, not
just TrackEvent. We will focus on TE here, even though the solution can be
easily extended to DS.

Goal

We want to allow developers to extend TrackEvent protos without requiring
multi-repo changes, to reduce the friction of using perfetto tracing to the
lowest possible.

More context

When extending tracing protos, there are two parties involved that depend on
that proto:

1. The writer on the caller-side

E.g., .cc/java files in system_server, which emits the trace point.
This needs to depend on the proto to get the per-field constants used to invoke
SDK methods, e.g.

  PerfettoTrace.instant(PerfettoTrace.JOB_SCHEDULER_CATEGORY_V3, eventName)
    .beginProto()
    .beginNested(JOB_SCHEDULER_JOB_FIELD_ID);

That JOB_SCHEDULER_JOB_FIELD_ID is autogenerated from the .proto definitions.

This requires little discussion: somehow the .proto file must live or be reachable
in the Android tree. The only discussion is where the source-of-truth lives
and who has a copy.

2. Trace Processor on the consumer side

TraceProcessor must be aware of the .proto descriptors so it can convert those
nested protos into structured args. That in turn allows both queries and
visualization in the UI.

Out Of Scope: TP custom parsing

The use-case of adding custom parsing C++ code to TraceProcessor to make some
advanced use of these protos is considered out of scope from this RFC. That one
requires far more than the descriptor and it is a niche use-case which, at
present, is not deemed to need a generalized solution.

The main challenge here is: how should TraceProcessor be aware of these
descriptors?

There are three possible solutions to this problem today, even before the
changes suggested by this RFC:

1. The protos are embedded in the TraceProcessor binary as they are available in
   the protos/ directory of upstream perfetto, on GitHub.

2. The protos get converted into a protobuf descriptor and that descriptor is
   made available in a Google-internal location. In turn those Google internal
   descriptors are available:

   1. In the Google-internal variants of trace_processor_shell
   2. In ui.perfetto.dev when opened from a Google-managed machine
   3. In internal Google dashboards (APC) which depend on the internal TP.

3. The descriptors can be embedded in the trace itself. See
   TracePacket.extension_descriptor and
   /docs/design-docs/extensions.md

Proposed solution

Hierarchical extension range registry

Extension field numbers in TrackEvent are managed via a hierarchical JSON
registry, analogous to IP subnet delegation. Each level of the hierarchy
delegates sub-ranges to child projects, which can further sub-delegate.

In track_event.proto, a single consolidated range is declared:

// Extension range for typed events defined in external projects.
// See track_event_extensions.json for the allocation registry.
extensions 1000 to 10000;

The allocation details live in a companion JSON file,
track_event_extensions.json, colocated with track_event.proto:

{
  "range": [1000, 10000],
  "allocations": [
    {
      "name": "chromium",
      "range": [1000, 1999],
      "contact": "somebody@chromium.org",
      "description": "Chrome browser and chromium-based projects",
      "repo": "https://chromium.googlesource.com/chromium/src",
      "proto": "base/tracing/protos/chrome_track_event.proto"
    },
    {
      "name": "android",
      "range": [2000, 2999],
      "contact": "somebody@android.com",
      "description": "Android OS platform",
      "registry": "protos/perfetto/trace/android/track_event_extensions.json"
    },
    {
      "name": "unallocated",
      "range": [3000, 9899]
    },
    {
      "name": "perfetto_tests",
      "range": [9900, 10000],
      "contact": "noreply@perfetto.dev",
      "description": "Reserved for Perfetto unit and integration tests",
      "proto": "protos/perfetto/trace/test_extensions.proto"
    }
  ]
}

Each allocation entry has:

• name — Short identifier for the range owner.
• range — Inclusive [start, end] field number range.
• ranges — Like range but supports scattered lists, e.g. [[1,9], [20,29]]. Required to handle migrations.
• contact — (optional) Owner email or group.
• description — (optional) What this range is for.
• repo — (optional) Git repo URL. If omitted, paths are local to this repo.
• Either proto (leaf: a .proto file defining extensions) or registry
  (intermediate: another .json file with further sub-delegation).
• unallocated entries have only name and range.

Sub-registries follow the same format. For example, Android's registry might
further delegate to frameworks/base, frameworks/av, etc., each owning a
sub-range of [2000, 2999].

Source-of-truth protos live all over the Android tree

We break down the current android_track_event.proto
into per-repo extension files:

$ANDROID/frameworks/base/proto/src/tracing/android_frameworks_base.proto
$ANDROID/frameworks/av/proto/src/tracing/android_frameworks_av.proto

and so on.

Each .proto follows the protozero wrapper-message pattern:

message AndroidTrackEvent {
  // Uses range [2000, 2099] per track_event_extensions.json.
  extend TrackEvent {
    optional string binder_service_name = 2001;
    optional string binder_interface_name = 2002;
    optional string apex_name = 2003;
    ...
  }
}

The only upstream (GitHub) change needed for a new top-level project is adding
an entry to track_event_extensions.json. This happens rarely.

Generating a fused descriptor

A C++ tool, gen_proto_extensions, reads the root JSON registry, recursively
follows local registry: references, compiles all referenced .proto files
using protoc_lib, validates that extension field numbers fall within their
declared ranges, and outputs a merged FileDescriptorSet proto.

gen_proto_extensions \
  --json protos/perfetto/trace/track_event/track_event_extensions.json \
  -I . \
  --out /tmp/extensions.descriptor \
  [--gzip]

The output only includes the subset of descriptor fields defined in Perfetto's
own protos/perfetto/common/descriptor.proto to minimize size.

At the build-level, in Android, an Android.bp rule takes the output of this
tool and makes sure this descriptor ends up in the system image, e.g. under
/system/usr/share/tracing_protos.descriptor.gz

Realistically we will need something similar for /vendor.

Traced dumps the protos in the trace

For Android builds (both standalone and in-tree), we can teach traced to simply
dump /system/usr/share/tracing_protos.descriptor.gz into the trace itself
so every trace collected from Android has all the up-to-date descriptors which
allow UI, APC, and any trace_processor version to JustWork.

This would happen in TracingServiceImpl::ReadBuffers() next to the various
MaybeEmit... so it is not affected by any buffering issue.

This opens a little problem: as these protos keep growing, there is a fixed
overhead (in terms of trace size) added to every Android trace. Let's pull some
numbers:

Even if we assume that the Android tracing protos grow as much as
atoms.proto, a monster-proto of 26K lines, the whole
atoms.descriptor is still 326 KB after gzip compression (252 KB after xz/lzma
compression). Realistically we have some years until this becomes a problem.

I can envision two solutions to mitigate this problem

Dump only the delta from last AOSP push (preferred)

Every time Android does AOSP drops we can copy back the various
frameworks_xxx_protos into GitHub. Then we need to make the dumping code slightly
more sophisticated and only dump the protos that changed from the last AOSP push.

Dump only the protos used

Today TracingServiceImpl already iterates through each packet at readback-time
for PacketStreamValidator. We could teach PacketStreamValidator to remember the
field IDs of the packets used (it will need to recurse one level into
TrackEvent) and only emit the descriptors for the packets used.
Realistically I don't think it's worth going into this complexity.

Note that we also have the ability of stripping out unwanted descriptors via the
privacy filter bytecode.

Other notes

• We will still need a copybara from Android -> google-internal for the logs proto

Alternatives considered

Use the Google-internal protos

Rather than dumping the descriptors into the trace, we could rely on the fact
that the various Android protos could be periodically merged into google3 and
become part of the Google-internal descriptors that the UI, APC and the internal
release of trace_processor_shell uses.

PRO: This avoids the increase in the trace size.

CONs: This doesn't JustWork(TM). There are a number of edge cases that are
not captured by this solution and would cause too many surprise factors:

1. Local development cycles: I am an Android developer, add a frameworks/base
   proto in my local tree and want to see how a trace looks like.
2. I am an Android engineer and I am opening a trace on a mac laptop using the
   upstream trace_processor_shell. People can't get easily ahold of a
   google-internal trace_processor_shell on mac laptops (this would require our
   team maintaining a google-internal mac build of tp_shell)
3. Lab test scenarios that are not using the google-internal TP build.
4. OEMs/Android partners creating their own proto extensions.
5. OEMs/Android partners opening Android traces from recent versions of Android (This could be solved with the RFC0006 but add yet another problem to solve)

---

💬 Discussion Guidelines:

• This discussion is automatically synced with the RFC document
• Please provide constructive feedback and suggestions

[github-actions[bot]]

📝 RFC Document Updated

View changes: Commit History

[primiano]

@betasheet as I think this could be interesting for chromium.
I think in chromium we can add a public method to TracingServiceImpl of the form RegisterProtoDescriptors(bytes/fd)

[betasheet]

Chromium currently relies on the extensions mechanism that was never fully finalized -.-

We commit extension messages into chromium, write the extension descriptor into the trace for local tracing, but rely on propagating the extensions across repos for background tracing. I'm not sure if Chrome would want the overhead of adding the descriptors to each trace there. I recall that some of the trace size/buffer limits are (historically) in the few hundred kBs.

[LalitMaganti]

#4324 is the bug tracking extension support. Planning on fixing it in the coming weeks.

[github-actions[bot]]

📝 RFC Document Updated

View changes: Commit History