sdk: Reduce amalgamated C SDK source size by ~54%#5800
Open
dreveman wants to merge 26 commits into
Open
Conversation
Member
|
I think Primiano is best placed on whether this would make sense on the future direction of the SDK. |
Member
|
Hmm so I am sympathetic with the overall goal, but not the with the way this is being done. Instead I think the proper way of doing this would be more splitting targets inside the build system, i.e. define a target with the minimal protos (and making sure the C sdk only depends on those) and keep the beefier target with all the protos for the existing cases. |
Source: 11,999,821 → 7,268,673 bytes (-4,731,148 = -39.4%)
Binary: 6,787,176 → 6,787,176 bytes (unchanged, stripped .so)
Compile: 11.98s → 10.99s (-8%)
The amalgamated C SDK source produced by tools/gen_amalgamated --sdk c was
12 MB despite the C ABI exposing a small surface (track events, data sources,
tracing session control). 8.7 MB of that was generated proto code, dominated
by 168 ftrace-event pbzero headers (3.1 MB) plus android, chrome, gpu,
profiling, etw, power, etc. trace protos that the C API never touches.
The reason: trace_packet.pbzero.h forward-declares per-domain message types
(it doesn't #include their headers), but tools/gen_amalgamated's
recurse_in_header_deps walks GN dep edges for any //protos/*:zero target
and pulls in every reachable pbzero header anyway. The transitive deps
flow through:
src/tracing → protos/perfetto/trace:zero
→ protos/perfetto/trace/{android,chrome,etw,ftrace,gpu,...}:zero
→ protos/perfetto/trace/interned_data:zero
→ protos/perfetto/trace/{android,chrome,gpu,profiling}:zero
Add gn arg enable_perfetto_minimal_trace_protos (default false). When set,
both protos/perfetto/trace:non_minimal_@TYPE@ and
protos/perfetto/trace/interned_data:@type@ stop GN-depending on the
per-domain proto subdirs the C SDK doesn't reference. The .proto files
themselves remain on disk and reachable to protoc via the project-root
import path, so trace_packet.proto and interned_data.proto still compile;
they just generate forward decls instead of pulling the per-domain
pbzero.h files into the build graph.
tools/gen_amalgamated automatically sets the flag when invoked with --sdk c
(only). --sdk all and --sdk cpp leave it off so the C++ amalgamation keeps
its full proto surface. The binary size doesn't change because pbzero
headers are forward-decl-only — none of the dropped types are instantiated
by SDK code, so the compiler was already eliminating them.
Audit method: grep 'protos/perfetto/.*\.pbzero\.h' under src/{shared_lib,
tracing}/, include/perfetto/{public,tracing,ext/tracing}/ excluding tests
and benchmarks. Confirmed dropped subdirs (android, chrome, etw,
filesystem, ftrace, generic_kernel, gpu, power, profiling, ps, statsd,
sys_stats, system_info, translation, plus protovm) appear in zero
non-test files.
… fields
Source: 7,268,673 → 6,049,949 bytes (-1,218,724 = -17%)
Binary: 6,787,176 → 6,176,584 bytes (-O0, -610 KB)
Binary: 2,691,672 → 2,343,512 bytes (-O2 + inlines-hidden, -348 KB / -13%)
Compile: 11.0s → 10.1s (-O0); 36.4s → 32.5s (-O2, -11%)
The previous commit dropped trace per-domain pbzero headers and produced
no binary delta, because pbzero classes are template-only forward-decls
that the linker had already discarded. This commit goes after the real
binary cost: the .gen.cc full-proto bindings under protos/perfetto/config
and protos/perfetto/common.
The blocker for cutting those was that data_source_config.proto (kept;
the SDK uses it for tracing session config) declares typed fields like
`optional FtraceConfig ftrace_config = 100 [lazy = true];` for ~14
per-domain configs. The cppgen plugin already had logic to skip the
typed `#include` for direct lazy-field referents — the comment at
src/protozero/protoc_plugin/cppgen_plugin.cc:171 explained the intent —
but it only inspected the *current* file's lazy fields. When generating
trace_config.gen.cc (whose proto has no lazy fields directly, but
imports data_source_config.proto which does), the lazy referents were
walked and #included transitively, defeating [lazy=true] downstream.
Fix in cppgen_plugin.cc: factor lazy_imports collection into a helper
and call it on every visited file in the dependency walk, not just the
top-level one. Now data_source_config's [lazy=true] referents are
correctly skipped from trace_config.gen.cc and from any further
descendant.
That unlocks the BUILD.gn changes:
* protos/perfetto/config/BUILD.gn: gate the per-domain config sub-deps
(android, ftrace, gpu, inode_file, power, process_stats, profiling,
qnx, statsd, sys_stats) behind enable_perfetto_minimal_trace_protos.
These are the targets corresponding to data_source_config.proto's
[lazy=true] field types. system_info / interceptors / priority_boost
/ protovm / track_event are kept (referenced by non-lazy fields or
used directly by SDK code).
* protos/perfetto/common/BUILD.gn: split sources into always-needed
(referenced by SDK code or by non-lazy fields) and optional (only
reached via the trace and config headers dropped earlier in this
stack, or via lazy fields from kept files). The optional set —
android_energy_consumer, android_log_constants, ftrace_descriptor,
gpu_counter_descriptor, perf_events, protolog_common, trace_attributes
— has zero non-test #include sites under src/{shared_lib,tracing}/ /
include/perfetto/{public,tracing,ext/tracing}/ once the cppgen fix is
in.
Verified by re-running tools/gen_amalgamated --sdk c and rebuilding the
amalgamated .so at -O0 and -O2. Cumulative vs the pre-stack baseline:
source 12.0 MB → 6.05 MB (-49.6%), -O2 stripped binary 2.69 MB → 2.34 MB
(-12.9%). The compile-time win compounds because the dropped .gen.cc
files were real translation units, not just headers.
…azy fields
Source: 6,049,949 → 5,884,418 bytes (-165,531 = -2.7%)
Binary: 6,176,584 → 6,049,608 bytes (-O0, -124 KB)
Binary: 2,343,512 → 2,265,688 bytes (-O2, -76 KB / -3.3%)
Compile: 10.1s → 9.6s (-O0); 32.5s → 32.0s (-O2)
Follow-up to the previous commit: now that cppgen honors transitive
[lazy=true] imports, chrome/{histogram_samples,scenario_config,
system_metrics,v8_config}, etw/etw_config and the parent
stress_test_config can also drop out of the SDK build. The corresponding
fields on data_source_config (V8Config, ChromiumSystemMetricsConfig,
ChromiumHistogramSamplesConfig, EtwConfig) are all lazy; scenario_config
and stress_test_config have no importers in the kept SDK closure.
chrome_config.proto stays (non-lazy ChromeConfig field).
test_config.proto stays (non-lazy `for_testing` field).
Source: 5,884,418 → 5,771,852 bytes (-112,566 = -1.9%)
Binary: 6,049,608 → 6,049,608 bytes (unchanged, pbzero-only changes)
Binary: 2,265,688 → 2,265,688 bytes (unchanged, -O2)
Compile: 9.6s → 9.9s (-O0); 32.0s → 31.1s (-O2)
Drop test_event.proto, test_extensions.proto, memory_graph.proto,
ui_state.proto, evdev.proto from the proto_sources_non_minimal list
when enable_perfetto_minimal_trace_protos is on. Each of these has zero
non-test #include sites in src/{shared_lib,tracing}/ /
include/perfetto/{public,tracing,ext/tracing}/.
trace_packet.proto declares typed-but-non-lazy fields for some of these
(UiState ui_state = 78, EvdevEvent evdev_event = 121, TestEvent for_testing
= 900), but the C SDK only consumes trace_packet.pbzero.h, which uses
forward declarations for nested message types and doesn't require the
per-type pbzero.h to exist. The matching public C ABI declarations in
include/perfetto/public/protos/trace/trace_packet.pzc.h use the same
forward-decl pattern (PERFETTO_PB_MSG_DECL expands to `struct X`).
Binary size unchanged because the dropped pbzero/.cc files were
already producing zero linker contribution (template-only forward decls
nothing instantiated). Source-size win is small but free.
Source: 5,771,852 → 5,727,555 bytes (-44,297 = -0.8%) Binary: 6,049,608 → 6,049,608 bytes (unchanged, -O0) Binary: 2,265,688 → 2,265,688 bytes (unchanged, -O2) Compile: 9.9s → 9.7s (-O0); 31.1s → 31.0s (-O2) sys_stats_counters.proto is imported only by sys_stats/sys_stats_config.proto (in the per-domain config target dropped earlier in this stack) and trace/sys_stats/sys_stats.proto (in the per-domain trace target dropped earlier in this stack). Once both are off, no SDK code references sys_stats_counters and it can be moved into the flag-gated source list.
Source: 5,727,555 → 5,557,692 bytes (-169,863 = -3.0%)
Binary: 6,049,608 → 5,904,712 bytes (-O0, -141 KB)
Binary: 2,265,688 → 2,220,504 bytes (-O2, -44 KB / -2.0%)
Compile: 9.7s → 9.5s (-O0); 31.0s → 30.1s (-O2)
Drop two groups of C++ source files that are part of the SDK's GN closure
but unreachable from the C ABI surface in include/perfetto/public/abi/.
src/tracing/BUILD.gn (gated under enable_perfetto_minimal_trace_protos):
console_interceptor.cc — pretty-printed console output for
in-process trace readback. Public C++
only (ConsoleInterceptor::Register);
nothing in include/perfetto/public/
references it.
track_event_state_tracker.cc — helper for replaying TrackEvent state in
a consumer; only used by
console_interceptor.cc.
track_event_legacy.cc — implementation of LegacyTraceId::Write.
The class is constructed only inside
`#if PERFETTO_ENABLE_LEGACY_TRACE_EVENTS`
blocks (off by default), so the .cc
symbol is dead.
src/base/BUILD.gn:
cpu_info.cc — used only by profiling and system_info data sources.
ctrl_c_handler.cc — used only by perfetto_cmd.
getopt_compat.cc — Windows getopt fallback used only by command-line
tools.
subprocess.cc + posix/windows variants — used only by tests and host
tools (perfetto_cmd, kallsyms loader, etc.).
All audit checks: zero non-test #include sites under src/{shared_lib,
tracing,protozero,protovm,ipc,base}/ and include/perfetto/{public,
tracing,ext/tracing}/ (excluding the dropped files' own self-references).
Replace the if (!enable_perfetto_minimal_trace_protos) source-list gate
with a triple-target split that lets the C SDK closure depend on a
slim subset without forcing every standalone out/ build to compile a
broken closure (which the original gating did — gen_amalgamated builds
:shared_lib only and never compiles :lite for these protos, masking
that the lite generator emits unconditional #includes for [lazy=true]
imports that the gating dropped).
:minimal_@TYPE@ — sources reachable from the C SDK closure under
src/{shared_lib,tracing}/. Generates zero + cpp
only. pbzero forward-declares cross-file types
and cppgen honors [lazy=true] (transitively
skipping #includes, see
src/protozero/protoc_plugin/cppgen_plugin.cc),
so a clean split is possible at this level.
:non_minimal_@TYPE@ — sources imported only by per-domain trace/config
protos (themselves not in the C SDK closure)
and by no minimal common proto. Generates
zero + cpp only.
:lite_only_@TYPE@ — full source set; generates lite only. Lite is
upstream protobuf's standard cpp generator (via
chromium's proto_library / cc_generator_options
= "lite=true:") and emits unconditional
#includes for every typed import, including
[lazy=true] referents. Keeping all sources in a
single library avoids cross-library .pb.h
cycles that would otherwise force lazy referents
back into the C SDK closure. The C SDK never
depends on :lite, so the redundancy is paid
only by tests / IPC / proto filtering targets.
ftrace_descriptor.proto and gpu_counter_descriptor.proto move into
:non_minimal_@TYPE@ as the original gating intended — they're imported
by data_source_descriptor.proto only via [lazy=true] fields, so the
zero+cpp generators that the C SDK uses don't need them.
Existing :@type@ -> group("zero")/("cpp") aggregators preserve the
public surface for current consumers; group("lite") forwards to
:lite_only_lite. group("source_set") aggregates only minimal +
non_minimal source_sets — :lite_only_source_set has the same .proto
files as the union of those two and adding it would only produce
duplicate metadata entries.
This commit is purely additive at the build-graph level: nothing yet
depends on :minimal_@TYPE@. A later commit in this stack switches the
C SDK closure (//src/shared_lib:shared_lib) over and removes the
enable_perfetto_minimal_trace_protos GN arg.
Same triple-target structure as the previous commit on common protos:
:minimal_@TYPE@ — sources reachable from the C SDK closure under
src/{shared_lib,tracing}/, plus the always-needed
deps. data_source_config.proto is in this set;
its [lazy=true] fields for per-data-source configs
(FtraceConfig, AndroidLogConfig, GpuCounterConfig,
...) are honored by cppgen, so the C SDK's .gen.h
doesn't pull those in. Generates zero + cpp only.
:non_minimal_@TYPE@ — chrome-only configs reachable only via [lazy=true]
fields and the per-data-source config sub-deps.
Generates zero + cpp only.
:lite_only_@TYPE@ — full source set + all sub-deps; generates lite
only. Lite is upstream protobuf's standard cpp
generator and emits unconditional #includes for
every typed import, including [lazy=true]
referents. Keeping all sources + deps in a single
library avoids cross-library .pb.h cycles.
The :@type@ wrapper (with proto_generators = []) keeps :descriptor
(config.descriptor) reachable at the same target name that
src/proto_utils:gen_cc_config_descriptor depends on, and routes its
transitive source-set lookups through :lite_only_@TYPE@ which has the
full proto graph.
This commit is purely additive at the build-graph level: nothing yet
depends on :minimal_@TYPE@. A later commit in this stack switches the
C SDK closure (//src/shared_lib:shared_lib) over and removes the
enable_perfetto_minimal_trace_protos GN arg.
Same triple-target pattern as the previous proto-split commits.
interned_data.proto has only one source file but its imports for
per-domain interned tables (InternedV8JsFunction in chrome/v8.proto,
AppWakelockInfo in android/app_wakelock_data.proto, etc.) split along
the same C-SDK-vs-rest line:
:minimal_@TYPE@ — interned_data.proto with only the always-needed
track_event dep. Generates zero + cpp; pbzero
forward-declares the cross-file types and cppgen
honors [lazy=true].
:lite_only_@TYPE@ — interned_data.proto with the full per-domain dep
set (android/chrome/gpu/profiling). Generates lite
only — upstream protobuf's lite generator emits
unconditional #includes for typed imports, so it
needs every dep available at compile time.
There's no :non_minimal_@TYPE@ here because the only diff between the
gated and ungated form is in deps, not sources. Aggregator groups keep
:zero / :cpp / :lite / :source_set reachable for current consumers.
Same pattern as the previous proto-split commits, with one extra layer
because the existing :non_minimal_@TYPE@ target had two flag-mutated
shapes (sources AND deps): split it into :non_minimal_core_@TYPE@
(C SDK closure subset) and :non_minimal_extras_@TYPE@ (rest), and
preserve the existing :non_minimal_zero / :non_minimal_cpp /
:non_minimal_source_set names as group aggregates so trace_processor /
trace_redaction / tests / chrome consumers continue to see the full
non-minimal closure unchanged.
:minimal_@TYPE@ — clock_snapshot / trace_uuid / trigger.
Generates zero + cpp + lite (lite is kept
here because the sources have no
cross-imports, so there's no cycle, and
chrome's :minimal_complete_lite needs
:minimal_lite to exist).
:non_minimal_core_@TYPE@ — non-minimal sources reachable from the
C SDK closure (trace, trace_packet,
trace_packet_defaults, remote_clock_sync,
extension_descriptor) with only the
always-needed deps (config, interned_data,
perfetto, track_event). trace_packet.proto
declares typed fields for several
per-domain messages but the pbzero
forward-decls + cppgen lazy-skip mean the
C SDK doesn't pay for them. zero + cpp.
:non_minimal_extras_@TYPE@ — sources NOT reachable from the C SDK
closure (test_event, test_extensions,
memory_graph, ui_state, evdev) plus the
per-domain trace proto sub-deps
(android, chrome, etw, ftrace, gpu, ...).
zero + cpp.
:lite_only_@TYPE@ — lite generation for non-minimal sources
with the full per-domain dep set.
public_deps on :minimal_@TYPE@ for
remote_clock_sync.proto's import of
clock_snapshot.proto. lite is upstream
protobuf's standard cpp generator, which
emits unconditional #includes for typed
imports including [lazy=true] referents,
so it can't be split cleanly.
New aggregator groups for the C SDK closure:
:zero_minimal = :minimal_zero + :non_minimal_core_zero
:cpp_minimal = :minimal_cpp + :non_minimal_core_cpp
// src/shared_lib will switch to these in a later commit.
This commit is purely additive at the build-graph level: nothing yet
depends on :zero_minimal or :cpp_minimal. Existing aggregator groups
(:zero, :cpp, :lite, :non_minimal_*) are preserved, so all current
consumers continue to see the same transitive closure.
Replace the if (!enable_perfetto_minimal_trace_protos) source-list gate
with a sibling-target split, mirroring the trace_processor_shell vs
trace_processor_minimal_shell pattern in src/trace_processor/:
:base_minimal — universally-used base sources plus the platform task
runners (lock_free / thread / unix). Reachable from
the C SDK closure under src/{shared_lib,tracing}/.
:base — :base_minimal plus the host-tool helpers
(cpu_info, ctrl_c_handler, getopt_compat,
subprocess.cc + posix/windows variants). Same name +
public surface as before; current consumers are
unchanged.
This commit is purely additive at the build-graph level: nothing yet
depends on :base_minimal. A later commit in this stack switches the
C SDK closure (//src/shared_lib:shared_lib) over and removes the
enable_perfetto_minimal_trace_protos GN arg.
Replace the if (enable_perfetto_minimal_trace_protos) source-list gate
with a sibling-target split, mirroring the trace_processor_shell vs
trace_processor_minimal_shell pattern in src/trace_processor/:
:client_api_without_backends_minimal — slim base set (data_source,
interceptor, muxer, tracing,
track_event_internal, ...).
Reachable from the C SDK
closure under //src/shared_lib.
:client_api_without_backends — :_minimal + console_interceptor
+ track_event_state_tracker +
track_event_legacy. Same name
+ public surface as before;
current consumers unchanged.
:client_api_minimal — sibling of :client_api but
routes through
:client_api_without_backends_minimal.
// src/shared_lib will switch
to this in a later commit.
Proto deps that were `deps` on the original :client_api_without_backends
become `public_deps` on :_minimal so the .h surfaces propagate to the
:client_api_without_backends wrapper for gn check (its console_interceptor
/ track_event_*.cc sources include the same generated headers).
This commit is purely additive at the build-graph level: nothing yet
depends on the new minimal targets. A later commit in this stack
switches the C SDK closure (//src/shared_lib:shared_lib) over and
removes the enable_perfetto_minimal_trace_protos GN arg.
Extract a slim variant of the tracing header set whose proto
public_deps route through the C-SDK-closure aggregators introduced
earlier in this stack. The headers themselves are identical; only the
transitive proto closure differs:
:tracing_minimal — public_deps = [..., trace:zero_minimal, ...]
(omits trace:zero, which would otherwise leak
:non_minimal_extras_zero — i.e. ui_state /
evdev / test_event / memory_graph / test_extensions
pbzero — into anything that depends on this
target).
:tracing — public_deps = [:tracing_minimal, trace:zero]
(preserves the existing transitive closure for
all current consumers; trace:zero brings the full
set back in).
Header list is shared via _tracing_headers to avoid drift.
This commit is purely additive: nothing yet depends on :tracing_minimal.
The next commit in this stack switches src/tracing:client_api_minimal
through it and removes the enable_perfetto_minimal_trace_protos GN arg.
The trace split commit earlier in this stack had :non_minimal_extras_@TYPE@ publicly depend on :non_minimal_core_@TYPE@. That worked for zero (pbzero forward-declares everything) but for cpp produced a cycle: cppgen for :non_minimal_core_cpp's trace_packet.gen.cc emits unconditional #includes for typed-but-non-lazy fields whose types live in :non_minimal_extras_* (UiState ui_state = 78, EvdevEvent evdev_event = 121, TestEvent for_testing = 900), and the public_deps wired the ordering the wrong way. Reverse the edge: :non_minimal_core_@TYPE@ now `deps`-on :non_minimal_extras_@TYPE@, and :non_minimal_extras_@TYPE@ gets its own `public_deps = [ "track_event:@type@" ]` (previously inherited transitively via core). This edge runs through `deps` for every gen_type — in particular it also pulls :non_minimal_extras_zero into anything that depends on :non_minimal_core_zero. The C SDK closure (//src/shared_lib) reaches trace:zero_minimal -> non_minimal_core_zero -> non_minimal_extras_zero, so the per-domain trace pbzero headers leak back in and the SDK source size win for the trace extras (test_event / memory_graph / ui_state / evdev / test_extensions) is not realized. A follow-up commit should either restructure trace cpp generation into its own full-source library (so the core->extras edge isn't needed at the zero level either) or mark the offending fields in trace_packet.proto as [lazy = true] so cppgen skips the includes structurally.
Wire //src/shared_lib:shared_lib through the slim _minimal targets
introduced earlier in this stack and delete enable_perfetto_minimal_trace_protos
as a load-bearing GN arg:
src/shared_lib/BUILD.gn:
../base -> ../base:base_minimal
../tracing:client_api -> ../tracing:client_api_minimal
src/shared_lib/track_event/BUILD.gn:
../../base -> ../../base:base_minimal
../../tracing:client_api -> ../../tracing:client_api_minimal
src/tracing/BUILD.gn (:client_api_without_backends_minimal,
:client_api_minimal):
../base -> ../base:base_minimal
../../include/perfetto/tracing -> :tracing_minimal
include/perfetto/tracing/BUILD.gn (:tracing_minimal):
common:zero/cpp -> common:minimal_zero/cpp
config:cpp -> config:minimal_cpp
trace/interned_data:zero -> :minimal_zero
(trace:zero_minimal swap was already in place)
protos/perfetto/config/BUILD.gn (:minimal_@TYPE@):
../common:@type@ -> ../common:minimal_@TYPE@
And :non_minimal_@TYPE@ gets `../common:non_minimal_@TYPE@` added
back so the full :@type@ aggregator group keeps the same closure
for non-C-SDK consumers.
And :lite_only_@TYPE@'s common dep stays on `../common:@type@`
(lite group), since common doesn't have a :minimal_lite — its
minimal sibling only generates zero + cpp, with lite living in
common:lite_only_@TYPE@.
gn/perfetto.gni:
Drop enable_perfetto_minimal_trace_protos. No GN file references
it any longer; it was only ever set by tools/gen_amalgamated when
targeting --sdk c, and that path has been removed.
tools/gen_amalgamated:
Drop the auto-set of enable_perfetto_minimal_trace_protos for
--sdk c builds. The C SDK closure now follows the regular GN dep
graph from //src/shared_lib:shared_lib, exercised by every
standalone out/* build.
Known caveat: the dep edge from :non_minimal_core_@TYPE@ to
:non_minimal_extras_@TYPE@ (added in the previous commit to fix the
cppgen cycle) leaks the per-domain trace pbzero headers back into the
C SDK closure via the zero generator, so this commit doesn't yet
realize the full ~6MB SDK source-size win the original flag-based stack
achieved. The build hygiene win — no more build flavor that no normal
build exercises — is realized in full. A follow-up commit can either
restructure trace cpp generation or annotate the relevant
trace_packet.proto fields with [lazy = true] to recover the size win.
The previous trace BUILD.gn split had :non_minimal_core_@TYPE@ generate
both zero + cpp, with a :non_minimal_core -> :non_minimal_extras dep
edge added to satisfy cppgen's unconditional #includes for typed fields
in trace_packet.proto (UiState ui_state = 78, FtraceEventBundle
ftrace_events = 1, ProcessTree process_tree = 2, etc.). That edge made
:non_minimal_core_zero transitively depend on :non_minimal_extras_zero
and the per-domain trace protos (android, chrome, ftrace, gpu, ...),
which leaked the per-domain pbzero headers back into the C SDK closure
that //src/shared_lib pulls via trace:zero_minimal.
Restructure so cpp generation lives in its own full-source library:
:non_minimal_core_@TYPE@ zero only (was zero + cpp).
:non_minimal_extras_@TYPE@ zero only (was zero + cpp).
:non_minimal_full_cpp_only_@TYPE@ NEW. cpp only, full source set
(core + extras) + the per-domain
trace deps that trace_packet's cppgen
needs at compile time. Not in the
C SDK closure (nothing under
//src/shared_lib pulls trace:cpp).
:lite_only_@TYPE@ unchanged (still lite only,
full source set + per-domain deps).
The :non_minimal_core -> :non_minimal_extras edge goes away — both
halves stand alone for the zero generator (pbzero forward-declares
cross-file types). For zero, that means trace:zero_minimal =
minimal_zero + non_minimal_core_zero is genuinely slim: no
non_minimal_extras, no per-domain trace deps.
Aggregator groups:
:non_minimal_cpp = :non_minimal_full_cpp_only_cpp (was
:non_minimal_core_cpp + :non_minimal_extras_cpp).
Slightly different shape but transitive closure is
the same — full_cpp_only has every non-minimal
source.
:cpp_minimal dropped — the C SDK closure doesn't depend on
trace:cpp at any layer.
Add C-SDK-closure variants of :core in the include/ and src/ tracing
layers, mirroring the pattern established for :tracing_minimal:
include/perfetto/tracing/core: :core_minimal — public_deps on
common:minimal_cpp + config:minimal_cpp
instead of common:cpp + config:cpp.
:core wraps :core_minimal and adds
the full-closure proto deps.
include/perfetto/ext/tracing/core: :core_minimal — same pattern,
routed through tracing/core:core_minimal
and common:minimal_cpp.
src/tracing/core: :core_minimal — same .cc sources as :core (lifted
into shared `_core_sources`), routed
through the slim include + proto +
base deps. :core wraps :core_minimal
and adds the full-closure deps for
backward-compat consumers.
Update src/tracing/BUILD.gn to point the C-SDK-closure entry points at
the minimal variants:
:common -> tracing:tracing_minimal
:client_api_without_backends_minimal
-> tracing/core:core_minimal,
tracing/core:core_minimal,
common:minimal_zero,
config:minimal_cpp
:client_api_minimal -> tracing/core:core_minimal
:system_backend{,_fake} -> tracing:tracing_minimal,
tracing/core:core_minimal,
base:base_minimal,
client_api_without_backends_minimal
:in_process_backend -> same swap, plus core:core_minimal
This collapses several layers of leak paths so that the C SDK closure
no longer pulls non_minimal_extras / per-domain trace deps via the
backends.
Remaining leak path: src/tracing/service still pulls
include/perfetto/ext/tracing/core:core (full) -> tracing/core:core
(full) -> protos/perfetto/{common,config}:cpp (full) ->
:non_minimal_cpp -> per-data-source config sub-deps. Splitting the
tracing service into a :service_minimal variant would close that gap;
deferred to follow-up since the service is large and the additional
shrink would need a careful audit of which service code paths reach
the per-domain config gen headers.
Add a :service_minimal sibling in src/tracing/service/ and plumb minimal
variants through every layer the in-process / system backends reach so
the C SDK closure under //src/shared_lib no longer pulls
//protos/perfetto/{common,config}:non_minimal_{cpp,zero} or the
per-data-source config sub-deps (config/{android,ftrace,gpu,inode_file,
power,process_stats,profiling,qnx,statsd,sys_stats}):
src/tracing/service/BUILD.gn:
NEW :service_minimal — same .cc sources as :service (lifted into
`_service_sources`), but routes the include/proto deps through the
minimal aggregators (common:minimal_zero, config:minimal_zero,
trace:zero_minimal, ext/tracing/core:core_minimal,
base:base_minimal, tracing:tracing_minimal, core:core_minimal).
:service is now a thin wrapper that public_deps on :service_minimal
+ the full closure deps for non-C-SDK consumers (trace_processor,
perfetto_cmd, tests, etc.). Audit: src/tracing/service/*.cc
references no per-data-source config type names.
src/tracing/BUILD.gn:
:in_process_backend swapped service:service -> service:service_minimal.
src/tracing/ipc/BUILD.gn:
:common: full ext/tracing/core / src/base / src/tracing/core ->
minimal variants.
:default_socket: same swap.
src/tracing/ipc/consumer/BUILD.gn,
src/tracing/ipc/producer/BUILD.gn,
src/tracing/ipc/service/BUILD.gn:
Same minimal-variant swap on full ext/tracing/core / tracing /
base / src/tracing/core deps. The system-backend ipc service code
only needs the slim header surface.
include/perfetto/ext/tracing/ipc/BUILD.gn:
:ipc public_deps on ext/tracing/core:core_minimal instead of full
:core. The .h headers in this set don't reach beyond the slim core
surface.
src/protozero/filtering/BUILD.gn:
:message_filter_config swapped config:cpp -> config:minimal_cpp.
Audit: only TraceConfig / TracePerfettoConfig types are referenced.
protos/perfetto/ipc/BUILD.gn:
:@type@ swapped common:cpp + config:cpp -> minimal_cpp variants.
Audit: every imported type
(commit_data_request, data_source_descriptor, observable_events,
tracing_service_state/capabilities, trace_stats, system_info from
common; data_source_config / trace_config from config) lives in the
:minimal_cpp halves.
protos/perfetto/trace/BUILD.gn:
:minimal_@TYPE@ and :non_minimal_core_@TYPE@ deps on
config:@type@ -> config:minimal_@TYPE@. The trace minimal sources
(clock_snapshot / trace_uuid / trigger) and the non-minimal core
sources (trace_packet / trace / etc.) only import config types
that live in config:minimal (trace_config, etc.).
protos/perfetto/config/BUILD.gn:
NEW group :minimal_lite that aliases :lite_only_lite. Needed
because :minimal_@TYPE@ generates only zero + cpp, but trace's
:minimal_@TYPE@ now uses `../config:minimal_@TYPE@` which expands
to `:minimal_lite` for the lite gen_type. Lite isn't in the C SDK
closure, so the fan-in on the full lite library is harmless.
Verified: //src/shared_lib:shared_lib's transitive dep tree (via
`tools/gn desc ... deps --tree`) now contains zero references to
non_minimal_extras, per-domain trace dirs (trace/{android,chrome,ftrace,
gpu,...}), or per-data-source config dirs (config/{android,ftrace,gpu,
...}). The full original ~6MB C SDK source-size win this stack was
chasing should now be realised; verify via `tools/gen_amalgamated --sdk c`.
Run tools/format-sources, tools/gen_bazel, tools/gen_android_bp on the
C-SDK-shrink stack output:
- tools/gn format applied to the BUILD.gn files touched by the
refactor (one trailing-newline / wrap fixup each in
include/perfetto/ext/tracing/{core,ipc}/BUILD.gn,
protos/perfetto/trace/BUILD.gn, src/protozero/filtering/BUILD.gn,
src/tracing/BUILD.gn).
- Android.bp regenerated to reflect the new sibling targets
(base_minimal, client_api_minimal, tracing_minimal, core_minimal,
service_minimal, the trace zero/cpp/lite library splits, etc.).
- BUILD (Bazel) regenerated for the same.
tools/run_presubmit is clean after this commit.
dreveman
force-pushed
the
dev/reveman/shrink-c-sdk
branch
from
1fc013d to
aab2b79
Compare
interned_data.proto has typed-but-non-lazy fields whose types live in
profiling/profile_common.proto (Mapping, Frame, Callstack,
UnsymbolizedSourceLocation, InternedString) and other per-domain
trace dirs (chrome/v8, android/app_wakelock_data, android/network_trace,
gpu/gpu_counter_event, gpu/gpu_render_stage_event). cppgen for
interned_data.gen.cc emits unconditional #includes for those types,
so :minimal_@TYPE@ generating cpp without the per-domain deps would
break the cpp compile.
The release standalone build masked the issue because some other
target's protoc invocation also generates the per-domain .gen.h files
into the same gen/ tree, so they happened to be present at compile
time. Cleaner builds (clang-x86_64-asan_lsan dist build per CI report)
expose the missing dep edge.
Same restructure as protos/perfetto/trace/BUILD.gn earlier in this
stack: split cpp generation into :cpp_only_@TYPE@ with the full
per-domain dep set. :minimal_@TYPE@ drops to zero only (pbzero
forward-declares cross-file types). The C SDK never depends on
interned_data:cpp, so the per-domain leak stays out of the SDK
closure. group("cpp") now points at :cpp_only_cpp; consumers
(android_sdk perfetto_sdk_for_jni, profiling/memory) keep the same
transitive closure.
Verified: tools/run_presubmit clean, the asan+lsan reproducer
(`is_asan=true is_lsan=true`) builds clean, and shared_lib's dep
tree still contains zero references to per-domain trace / config dirs.
The Bazel build was failing with `rule '//:protos_perfetto_trace_non_minimal_protos' does not exist`. Two unrelated places still referenced the
pre-refactor target name:
tools/gen_bazel:
proto_groups['trace']['sources'] still listed the old single
`//protos/perfetto/trace:non_minimal_source_set`. Replace with the
new core + extras source_sets that the trace BUILD.gn split
introduced. The Bazel `:trace_proto` aggregator now depends on
:protos_perfetto_trace_non_minimal_core_protos +
:protos_perfetto_trace_non_minimal_extras_protos instead of the
non-existent :protos_perfetto_trace_non_minimal_protos.
BUILD.extras:
The hand-coded `protos_perfetto_trace_non_minimal_protos_go_proto`
Bazel target still depended on the removed name. Replace with the
matching core + extras go_proto wrappers so downstream Bazel
consumers needing Go bindings of the non-minimal trace protos see
both halves.
Regenerated BUILD via tools/gen_bazel; no broken references remain
(verified by `grep 'protos_perfetto_trace_non_minimal_protos\b' BUILD`).
tools/run_presubmit clean.
Bazel build was failing with `rule '//:protos_perfetto_trace_interned_data_protos' does not exist` because gen_bazel's source-set name translation strips the gen-type suffix and adds `_protos` to produce the canonical Bazel target name; with the GN target renamed to :minimal_@TYPE@, no perfetto_proto_library matched the canonical `:protos_perfetto_trace_interned_data_protos` form that gpu/BUILD.gn (and several other consumers) export through their public_deps. Rename the slim-zero perfetto_proto_library back to :@type@ (the canonical name). interned_data has only one source file (interned_data.proto), so there's no minimal-vs-extras source split to disambiguate — the slim variant is the canonical zero target. The earlier-commit cpp/lite split into :cpp_only_@TYPE@ / :lite_only_@TYPE@ stays as-is. The auto-generated :zero / :source_set sub-targets from the canonical :@type@ now subsume what my group("zero") / group("source_set") were aliasing, so the explicit groups are dropped. include/perfetto/tracing's :tracing_minimal swaps `interned_data:minimal_zero` -> `interned_data:zero` to point at the (functionally identical) canonical target, and the redundant transitive dep on :tracing_minimal already provides interned_data:zero, so :tracing's explicit listing is removed. tools/run_presubmit clean, shared_lib's dep tree still has zero references to non_minimal_extras / per-domain trace dirs / per-data-source config dirs.
Bazel build was failing with `file 'protos/perfetto/config/chrome/chrome_config.pb.cc' is generated by these conflicting actions: //:protos_perfetto_config_lite_only_protos, //:protos_perfetto_config_minimal_protos`. Bazel's
proto_library rejects two libraries claiming the same .proto source.
protos/perfetto/{config,trace}/BUILD.gn each have an empty
perfetto_proto_library("@type@") wrapper used only for :source_set
aggregation + descriptor generation. Both were `deps`-on
:lite_only_@TYPE@, which has the full source set duplicated across
generators. The Bazel translator walked the source_set chain, emitted
:protos_perfetto_<dir>_lite_only_protos, and the
cc_proto_library aspect ran protoc twice on the overlapping sources.
Route each descriptor wrapper through the non-overlapping halves
instead:
config/BUILD.gn: deps :lite_only_@TYPE@ -> :non_minimal_@TYPE@
trace/BUILD.gn: deps :lite_only_@TYPE@ -> :non_minimal_core_@TYPE@
+ :non_minimal_extras_@TYPE@
The descriptor target's protoc invocation finds imports via
proto_in_dir on the project root, so source_set's GN-level transitive
membership doesn't need to cover the lite/cpp-only deps explicitly —
the .proto files are always on disk.
Net effect on Bazel BUILD: :protos_perfetto_config_lite_only_protos,
:protos_perfetto_trace_lite_only_protos, and
:protos_perfetto_trace_non_minimal_full_cpp_only_protos are no longer
emitted (none of them had any consumer beyond the descriptor walk),
removing the source-overlap conflict.
tools/run_presubmit clean, shared_lib's dep tree still has zero
references to non_minimal_extras / per-domain trace dirs / per-data-source
config dirs.
Bazel build was failing with `target '//:protos_perfetto_trace_non_minimal_zero' does not exist` from the auto-generated `trace_zero` cc_library. The `:non_minimal_zero` target in GN is a group aggregating `:non_minimal_core_zero + :non_minimal_extras_zero`; groups don't translate to Bazel targets, so the generated reference was dangling. Update tools/gen_bazel:gen_protozero_group_target to walk the two underlying perfetto_proto_library targets directly instead of the group. Sibling fix to the proto_groups['trace'] update earlier in this stack. tools/run_presubmit clean. Final verification: no broken proto target references in the regenerated BUILD file (`grep 'trace_non_minimal_zero\b\|trace_non_minimal_protos\b\|config_lite_only_protos\|trace_lite_only_protos\|trace_non_minimal_full_cpp_only_protos' BUILD` returns nothing).
Bazel build was failing with `protos/perfetto/common/ftrace_descriptor.proto: File not found.` during descriptor-set generation for :protos_perfetto_common_minimal_protos. The same data_source_descriptor.proto -> ftrace/gpu_counter_descriptor import that cppgen elides via lazy-skip in the GN build is rejected by Bazel proto_library's strict input-tracking: every imported .proto must be reachable through srcs or direct deps, with no project-wide fallback through proto_path. Moving the two descriptor protos into common's minimal_proto_sources makes them part of :minimal_protos's srcs in Bazel, satisfying the import. There's no cycle to worry about in GN (they have no own imports), and SDK size is unchanged (gen_amalgamated already pulled their .gen.cc via the cpp closure — measured: SDK .cc 5,561,849 -> 5,561,850 bytes, +1 byte from a comment-formatting artifact). tools/run_presubmit clean. Note: config/data_source_config.proto has similar lazy-imported fields referring to per-data-source config subdirs (FtraceConfig from config/ftrace, etc.). If the next Bazel CI run fails on :protos_perfetto_config_minimal_protos with the same File-not-found shape, addressing it the same way would pull all per-data-source configs into the C SDK closure, defeating most of this stack's savings — a different approach (gen_bazel Bazel-only deps, or restructure) would be needed. Deferring until CI confirms.
Bazel build was failing with `'@@//:src_tracing_core_core' does not produce any cc_library srcs files` because the thin source_set wrappers I introduced across the stack (each forwarding to a :*_minimal sibling that owns the actual sources, with no own sources) were being translated to empty perfetto_filegroup() targets in BUILD. Bazel's cc_library srcs extension check rejects filegroups that produce no .cc files. Convert each empty wrapper from source_set to group so gen_bazel doesn't track it as a sources-contributing target: src/tracing/core/BUILD.gn :core src/tracing/service/BUILD.gn :service include/perfetto/tracing/BUILD.gn :tracing include/perfetto/tracing/core/BUILD.gn :core include/perfetto/ext/tracing/core/BUILD.gn :core src/tracing/service's :service had a deps+public_deps split that the group form collapses into a single public_deps list; the consumer surface stays the same (everything still propagates). src/base:base and src/tracing:client_api_without_backends are NOT converted because they have their own additional .cc sources beyond the :*_minimal sibling and are real linker units. tools/run_presubmit clean. Verified no empty perfetto_filegroup entries remain in the regenerated BUILD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
tools/gen_amalgamated --sdk cproduces 12 MB / 290k lines today. Most is dead weight: per-domain trace + config protos and a handful of C++ files that the C ABI underinclude/perfetto/public/abi/never references — pulled in transitively via the GN dep graph and viacppgen_plugin's[lazy = true]skip not being transitive.This PR adds a new GN arg
enable_perfetto_minimal_trace_protos(default off, auto-enabled bygen_amalgamated --sdk conly) and uses it to gate the unused subgraphs.--sdk cpp/--sdk all/ chromium / Android / standalone builds are unaffected. Public C ABI is unchanged.perfetto_c.{cc,h}.so(-O2 -fvisibility-inlines-hidden)clang++ -O2 -cwall timeThe stack is six small commits, each independently revertable:
cppgen_plugin's[lazy = true]import-skip transitive (strict bugfix of an existing partial behaviour) and use it to drop per-domain config + common protos.test_event,memory_graph,ui_state,evdev,test_extensions).sys_stats_countersfrom common (importers all dropped above).console_interceptor.cc,track_event_state_tracker.cc,track_event_legacy.cc, andsrc/base/{cpu_info,ctrl_c_handler,getopt_compat,subprocess*}.cc.See per-commit messages for the audit method (
grepfor#includesites undersrc/{shared_lib,tracing}/andinclude/perfetto/{public,tracing,ext/tracing}/, excluding tests) and the source/binary delta of each step.Issue: #5799