enhancement(core): add helpers for spawning dynamic workers under component supervisors

[tobz]

Summary

This PR adds helpers to support easily spawning associated workers/child tasks under a component's dedicated supervisor in order to replace bespoke/one-off tasks spawned directly on the ambient runtime or global thread pool.

In prior PRs, we've moved topologies, and indeed the individual components within them, to be managed entirely via supervision trees. However, we stopped short of also managing the individual child tasks that some components will spawn. These are things like connection handlers, compute-heavy encoding/decoding, and so on.

This PR adds a new helper, ComponentSpawner, designed specifically for components to spawn associated workers/child tasks directly on their dedicated supervisor such that they become implicitly associated with the component (good for attribution) and share the same lifecycle as the component: when components are shut down, so are their associated tasks.

We explicitly are not porting over components to use this new helper in this PR, as there's a few different patterns of how components spawn tasks, and we'll tackle those in follow-up PRs as so more work will be required to support each pattern.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

New and existing tests.

References

DADP-2

[datadog-prod-us1-5]

More details

The new component-scoped spawner and supervisor shutdown budget are consistent with the existing supervision lifecycle: dynamic children inherit component ownership, ordered and concurrent drains remain bounded as documented, and nested supervisors retain control of their own shutdown. Static review found no concrete production regression in the changed paths.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 266a316 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: cee29055-d32d-4520-8d30-4baa7100ec71
Baseline: d9a74edd · Comparison: 048acaf3 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +0.55	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.38	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.34	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.04	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.04	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 228 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 52 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.3 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 378 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.9 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 266a31624d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Fix the malformed rustdoc link

This malformed intra-doc link is parsed as one link target that includes the or [ text, so API documentation generation for saluki-core fails once rustdoc checks links under the crate's denied warnings. Close the first link before or (or make these two separate links) so ChildBuilder's public docs build cleanly.

Useful? React with 👍 / 👎.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: d9a74ed · Comparison: 048acaf · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.55 MiB (baseline) vs 41.59 MiB (comparison)
Size Change: +32.36 KiB (+0.08%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
core	+18.07 KiB	1927
anon.fe5b072134a56fcc9e3faefbefac5a24.3.llvm.12329090102411404263	+17.53 KiB	1
anon.e254d0c82dc2bd8175b93ef7e3b1fb53.15.llvm.15624024362873184253	-17.53 KiB	1
anon.1d6497f9a04757386c21a0e9071d0246.666.llvm.6229194563851314955	-12.29 KiB	1
anon.fe5b072134a56fcc9e3faefbefac5a24.559.llvm.12329090102411404263	+12.11 KiB	1
agent_data_plane_config::shared::_	+12.00 KiB	16
agent_data_plane_config::domains::dogstatsd	-10.49 KiB	13
anon.c6a0d7b2701d7cf70c7a3da1770431f5.297.llvm.15420591089772377614	+9.90 KiB	1
anon.e254d0c82dc2bd8175b93ef7e3b1fb53.604.llvm.15624024362873184253	-9.80 KiB	1
serde_core	+9.36 KiB	89
anon.c6a0d7b2701d7cf70c7a3da1770431f5.576.llvm.15420591089772377614	+9.25 KiB	1
anon.e254d0c82dc2bd8175b93ef7e3b1fb53.19.llvm.15624024362873184253	-9.25 KiB	1
saluki_core::observability::metrics	+8.92 KiB	21
alloc	-8.59 KiB	226
anon.fe5b072134a56fcc9e3faefbefac5a24.1.llvm.12329090102411404263	+8.49 KiB	1
anon.58d7e071c6f28db3af2cf9de910d5af7.1.llvm.4032246748914016819	-8.40 KiB	1
saluki_core::runtime::supervisor	+7.93 KiB	133
agent_data_plane_config::_::_<impl serde_core	-7.23 KiB	1
anon.fe5b072134a56fcc9e3faefbefac5a24.11.llvm.12329090102411404263	+6.76 KiB	1
anon.58d7e071c6f28db3af2cf9de910d5af7.11.llvm.4032246748914016819	-6.67 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +42.5Ki  [NEW] +42.4Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h34d24cb52f54871d
  [NEW] +36.9Ki  [NEW] +36.7Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hfa69234aabe7888d
  [NEW] +35.1Ki  [NEW] +34.9Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hc9b51a05f244dad1
  [NEW] +26.9Ki  [NEW] +26.7Ki    saluki_core::topology::built::BuiltTopology::spawn_inner::_{{closure}}::h32cba20cc95dd159
  [NEW] +26.5Ki  [NEW] +26.3Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h16516a848199c4de
  [NEW] +20.4Ki  [NEW] +20.2Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::hd2c96dcd5b651c82
  [NEW] +20.0Ki  [NEW] +19.9Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::he372223795480ddf
  [NEW] +17.5Ki  [NEW]     +76    anon.fe5b072134a56fcc9e3faefbefac5a24.3.llvm.12329090102411404263
  [NEW] +17.4Ki  [NEW] +17.3Ki    saluki_core::runtime::worker_state::WorkerState::shutdown_workers::_{{closure}}::h32e43864973c7bd2
  [NEW] +17.3Ki  [NEW] +17.2Ki    saluki_core::runtime::supervisor::Supervisor::run_inner::_{{closure}}::h1feb57d02a233c04
  [NEW] +16.7Ki  [NEW] +16.5Ki    _<tracing::instrument::Instrumented<T> as core::future::future::Future>::poll::h8d34c4031cba6528
  +0.1% +5.55Ki  -0.0% -2.26Ki    [10987 Others]
  [DEL] -17.5Ki  [DEL]     -76    anon.e254d0c82dc2bd8175b93ef7e3b1fb53.15.llvm.15624024362873184253
  [DEL] -19.8Ki  [DEL] -19.7Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h6a552796290bab91
  [DEL] -20.4Ki  [DEL] -20.2Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h485ff2303b4557b3
 -98.1% -24.7Ki -98.5% -24.7Ki    agent_data_plane::state::metrics::rules::get_datadog_agent_remappings::h4ff388cc81845209
  [DEL] -25.9Ki  [DEL] -25.8Ki    saluki_core::topology::built::BuiltTopology::spawn_inner::_{{closure}}::haead73c1688408b0
  [DEL] -26.5Ki  [DEL] -26.3Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h1db26b4b2bc3c9f1
  [DEL] -35.1Ki  [DEL] -34.9Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h99e6e806e208f4e8
  [DEL] -38.1Ki  [DEL] -37.9Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hbcbb45ee17ef2879
  [DEL] -42.3Ki  [DEL] -42.2Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hb5fa11377218b0bc
  +0.1% +32.4Ki  +0.1% +24.1Ki    TOTAL

[tobz]

• enhancement(core): add helpers for spawning dynamic workers under component supervisors #2326 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.