[ABLD-464] Debug-symbol packages: packaging-time filter (Plan B) - #54944
Open
aiuto wants to merge 2 commits into
Open
[ABLD-464] Debug-symbol packages: packaging-time filter (Plan B)#54944aiuto wants to merge 2 commits into
aiuto wants to merge 2 commits into
Conversation
**what this does** Adds a packaging-time filter, `dd_pkg_files_stripped` (in `bazel/rules/dd_packaging/dd_pkg_strip_transform.bzl`), that replaces plain `pkg_files` calls for shipped binaries. It strips object files as they're collected into a package and splits the removed debug info into a parallel `name + "_debug"` sibling target, without requiring any cooperation from the binary's own build rule -- it works directly against prebuilt-file-backed targets like `@agent_binary//:agent`, which is this design's main advantage over the alternative provider/rule-based approach considered in the same design doc (Plan A, implemented separately). A single `_dd_strip_split` action runs strip/objcopy/dsymutil once per file and declares both the "stripped" and "debug" outputs at once; the public `dd_pkg_strip_transform` rule only projects one half of that shared result, so building both the normal package and its debug sibling never doubles the work. File-type detection (ELF/Mach-O/PE) happens at action-execution time in a new Python driver (`dd_strip_driver.py`), since Starlark can't inspect file contents during analysis; the Starlark side only applies a cheap extension/mode-bit heuristic to skip spawning actions for obviously non-binary files. Tool paths come from a new `//bazel/toolchains/dd_strip` toolchain: Linux and Windows source `strip`/`objcopy` off the existing `cc_toolchain`; macOS hardcodes `/usr/bin/strip` and `/usr/bin/dsymutil` (modeled after `//bazel/rules/rewrite_rpath`'s hardcoded-path pattern rather than `//bazel/toolchains/codesign`'s auto-detecting repository_rule, since that pattern generates a `config_setting` per tool and new config_settings need separate design review). Platform semantics match `omnibus-ruby/lib/omnibus/stripper.rb`: Linux does the 3-step objcopy --only-keep-debug / strip / objcopy --add-gnu-debuglink; macOS runs `strip -x` plus `dsymutil` into a `.dSYM` bundle; Windows ships a `strip`'d binary alongside the unstripped original (mingw's toolchain has no `objcopy`, so there's no split-DWARF story there, matching the existing `windows_symbol_stripping_file` behavior). Wired into `packages/agent/product/BUILD.bazel` (the four `dda_built_*_binary` targets, plus a new `all_files_debug` pkg_filegroup) and `packages/installer/windows/BUILD.bazel` (`installer_binary` plus a new `installer_components_debug`). Removed the one-shot `-Cstrip=symbols` rustc flag from `bazel/configs/system_probe_lite.bazelrc`, which used to destroy debug info before packaging ever saw it. Updated `packages/AGENTS.md` and `packages/installer/MIGRATION_PLAN.md` to reflect that symbol stripping is now migrated. **testing** Built and inspected output on macOS (arm64) at each stage: - A throwaway target wrapping `//cmd/loader:loader` confirmed both the "stripped" and "debug_only" modes produce sane output: `strip -x` removed local symbols, `dsymutil` produced a `.dSYM` bundle with a DWARF resource, and building both mode targets together registered the `DdStripSplit` action exactly once (verified in the verbose build log). - `bazel build //packages/agent/product:all_files //packages/agent/product:all_files_debug` and `//packages/installer/windows:whole_distro_tar //packages/installer/windows:installer_components_debug` succeed against the real prebuilt targets (`@agent_binary`, `@trace_agent_binary`, `@process_agent_binary`, `@privateactionrunner_binary`, `@installer_binary`). - Along the way, fixed two real bugs surfaced by this: `Args.add()` can't take a directory (needed `.path` for the `.dSYM` output), and the passthrough path crashed when the declared debug output was a directory but the driver couldn't recognize the input's format (hit when packaging a Linux ELF prebuilt from a macOS host, since the macOS toolchain has no objcopy -- this is a local-only condition, not a bug in the Linux path itself). - `bazel test //bazel/rules/dd_packaging:_dd_packaging_tests` (the existing suite) still passes -- no regression in the unrelated packaging rules this shares a package with. - Could NOT verify on this machine: the actual Linux ELF split (no Linux sandbox/CI available locally -- the `@trace_agent_binary` prebuilt is a real Linux ELF, but macOS's `dd_strip` toolchain has no `objcopy`, so it fell through to passthrough rather than exercising `_strip_elf`), and whether the produced `.dSYM`/`.debug` artifacts are actually useful to a debugger (no symbolication round-trip was attempted). - No unit tests were added for `dd_pkg_strip_transform`/`_dd_strip_split` themselves (analysistest-style, like `dd_packaging_test.bzl`) -- only manual `bazel build` + filesystem inspection. **next steps** - The Linux objcopy path needs validation in CI or a Linux sandbox before this can be trusted end-to-end. - The macOS `dSYM`-splitting approach hasn't been reviewed by the build team; `strip -x` semantics (keeps global symbols) match omnibus but deserve a second look. - Whatever builds `bin/agent/agent` and friends outside Bazel (dda/omnibus) must also stop pre-stripping, or there will be no debug info left for this filter to split off -- that's a separate, out-of-scope dependency for this PR (the Rust/system-probe-lite case is fixed here; the Go/dda case is not). - Plan A (implemented separately, in parallel) builds its own independent copy of the same toolchain shape under `//bazel/toolchains/dd_strip` by design -- deduplicating the two toolchains is a deliberate follow-up, not done here.
Preserves the ABLD-464 context and design rationale (both candidate strategies, platform semantics, and the macOS codesign-ordering issue found while implementing the sibling PR) alongside the code, since the original planning doc lives only on the local machine that planned it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
aiuto
added
changelog/no-changelog
aiuto
added
changelog/no-changelog
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1c002bc5e9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| executable. | ||
| """ | ||
| lower = dest.lower() | ||
| if lower.endswith(_ALWAYS_BINARY_SUFFIXES) or ".so." in lower: |
There was a problem hiding this comment.
endswith
When any dd_pkg_files_stripped target has at least one source, analysis reaches this call with _ALWAYS_BINARY_SUFFIXES as a tuple, but Bazel Starlark's string.endswith accepts a string rather than Python's tuple-of-suffixes form. Consequently the newly wired packaging targets fail during analysis before any strip action is created; test the suffixes individually instead.
Useful? React with 👍 / 👎.
| ) | ||
|
|
||
| stripped_dest_src_map[dest] = stripped_out | ||
| debug_dest_src_map[dest] = debug_out |
There was a problem hiding this comment.
When a _debug sibling is consumed by pkg_filegroup or an archive rule, PackageFilesInfo uses this dictionary key as the packaged destination, so mapping debug_out back to the original dest discards the declared .debug or .dSYM suffix. Linux therefore packages agent.debug as agent even though the stripped binary's GNU debuglink names agent.debug, macOS loses the bundle name, and Windows symbol extraction looking for .exe.debug finds nothing; use the debug artifact's intended suffixed destination here.
Useful? React with 👍 / 👎.
| # windows_symbol_stripping_file semantics). | ||
| dd_pkg_files_stripped( | ||
| name = "installer_binary", | ||
| srcs = ["@installer_binary//:installer"], |
There was a problem hiding this comment.
When the Windows Go build emits the companion <binary>.exe.pdb, this target captures only the executable, so installer_binary_debug can never contain the PDB. The existing symbol-store pipeline reads only .pdb entries from debug archives, and the Omnibus windows_symbol_stripping_file lifecycle explicitly relocates those companions, so switching this component to the Bazel debug target produces an incomplete or empty Windows symbol store; expose and collect the companion PDB alongside the unstripped executable.
AGENTS.md reference: packages/AGENTS.md:L221-L225
Useful? React with 👍 / 👎.
Contributor
Files inventory check summaryFile checks results against ancestor fec92d8c: Results for datadog-agent_7.84.0~devel.git.220.1c002bc.pipeline.131245139-1_amd64.deb:No change detected Results for datadog-iot-agent_7.84.0~devel.git.220.1c002bc.pipeline.131245139-1_amd64.deb:No change detected |
Contributor
Static quality checks✅ Please find below the results from static quality gates 33 successful checks with minimal change (< 2 KiB)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: fec92d8 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | quality_gate_logs | % cpu utilization | +1.06 | [+0.19, +1.94] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_idle | memory utilization | +0.21 | [+0.11, +0.31] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.18 | [-0.06, +0.43] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_no_fs_load | memory utilization | -0.03 | [-0.18, +0.11] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_security_mean_fs_load | memory utilization | -0.10 | [-0.17, -0.03] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.13 | [-0.17, -0.09] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_private_action_runner | memory utilization | -0.21 | [-0.33, -0.09] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | -0.37 | [-0.49, -0.25] | 1 | Logs bounds checks dashboard |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | quality_gate_idle | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 172.16MiB ≤ 178MiB | bounds checks dashboard |
| ✅ | quality_gate_idle | total_bytes_received | 10/10 | 739.13KiB ≤ 819.20KiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 4 = 4 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 521.42MiB ≤ 538MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | total_bytes_received | 10/10 | 1.12MiB ≤ 1.25MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 16 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 200.67MiB ≤ 229MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_logs | total_bytes_received | 10/10 | 264.27MiB ≤ 292MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 345.89 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 18 ≤ 40 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 414.53MiB ≤ 439MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | total_bytes_received | 10/10 | 0.94GiB ≤ 1.04GiB | bounds checks dashboard |
| ✅ | quality_gate_private_action_runner | memory_usage | 10/10 | 72.09MiB ≤ 76MiB | bounds checks dashboard |
| ✅ | quality_gate_security_idle | cpu_usage | 10/10 | 26.94 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_idle | memory_usage | 10/10 | 326.87MiB ≤ 335MiB | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | cpu_usage | 10/10 | 60.94 ≤ 200 | bounds checks dashboard |
| ✅ | quality_gate_security_mean_fs_load | memory_usage | 10/10 | 300.72MiB ≤ 314MiB | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | cpu_usage | 10/10 | 21.09 ≤ 100 | bounds checks dashboard |
| ✅ | quality_gate_security_no_fs_load | memory_usage | 10/10 | 312.58MiB ≤ 343MiB | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_security_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_idle, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_no_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_private_action_runner, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check total_bytes_received: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_security_mean_fs_load, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds
dd_pkg_files_stripped, a drop-in replacement forpkg_filesthatstrips object files as they're collected into a package and splits the
removed debug info into a parallel
<name>_debugsibling target — via a newdd_pkg_strip_transformrule operating on the already-mergedPackageFilesInfotree, and a shared//bazel/toolchains/dd_striptoolchain (Linux objcopy 3-step / macOS strip+dsymutil / Windows mingw
strip, matching omnibus's
Strippersemantics). File-type detection(ELF/Mach-O/PE) happens at action-execution time in a Python driver, since
Starlark can't inspect file contents during analysis.
Because this hooks in at the packaging layer rather than the binary's own
build rule, it works directly against today's
prebuilt_file.bzl-backedproduct binaries (
@agent_binary//:agent, etc.) with no changes needed tohow those targets are built — this is its main advantage over the
alternative provider/rule-based strategy in the companion PR. Wired into
packages/agent/product/BUILD.bazel(4 binaries + newall_files_debug)and
packages/installer/windows/BUILD.bazel.A single strip/split action produces both the "stripped" and "debug_only"
outputs per file, so building both the normal package and its debug sibling
never doubles the work.
See
dbg_symbol.mdfor full context, including a companion PR(branch
aiuto/454-a, other worktree) implementing an alternativeprovider/rule-based strategy for the same ticket, and a known macOS
codesign-ordering bug found while implementing that sibling PR.
Motivation
ABLD-464: Bazel packaging
has no equivalent to omnibus's stripped-binary + separate debug-symbol
("-dbg") package generation.
packages/AGENTS.mdandpackages/installer/MIGRATION_PLAN.mdboth track this as an open gap.Describe how you validated your changes
On macOS arm64 (no Linux sandbox available locally, so the Linux
objcopypath is unverified — needs CI; the macOS toolchain has no
objcopy, so areal ELF binary fell through to passthrough as expected):
action; a real
.dSYMbundle with DWARF content is produced;strip -xremoves local symbols.
bazel build //packages/agent/product:all_files,:all_files_debug,:whole_distro_tar, and//packages/installer/windows:installer_components_debugall buildsuccessfully against the real prebuilt binary targets.
_dd_packaging_testssuite still passes.Additional Notes
Known open items (see
dbg_symbol.md):objcopypath unverified locally — needs CI or a Linux sandbox.strip+dsymutilapproach has no prior art in omnibus or this repoand needs build-team confirmation before merge.
dda inv agent.build/omnibus) must not pre-strip them for either strategy to have DWARF left to
split — out of scope here, flagged as a cross-cutting dependency (Rust
case already fixed by removing
-Cstrip=symbols).aiuto/454-aPR is a deliberatefollow-up once one strategy is chosen.
(stripping after
rewrite_rpath's re-sign invalidates the signature)should be checked for here too once this path produces real signed macOS
binaries through
dd_cc_packaged-style consumers.