enhancement(core): add support for dynamically added child processes in Supervisor

[tobz]

Summary

This PR adds support for dynamically adding child processes to a Supervisor, along with other changes to better align our supervisor behavior with the behavior of the granddaddy of all supervisors, supervisor.erl in Erlang/OTP.

Currently, Supervisor is designed to be configured before running, and then, once running, only ever manage the workers it was configured with. This is sufficient for almost all of our usage, but it makes it impossible to support a small but reasonable use case: supervision of dynamically spawned tasks, like network connection handlers (DSD stream handlers, for example).

This PR introduces support for dynamically spawned workers, along with some other behavioral uplifts, to support a higher-fidelity implementation of supervisors:

• SupervisorHandle, a new type that can be used to dynamically spawn workers at runtime, when a supervisor is already running
• ShutdownMode, to support concurrently stopping processes during shutdown, which is useful in a world where a supervisor may be supervising many dynamic child processes
• support for "significant" children and automatic shutdown, which allows triggering a supervisor to shutdown completely if special child processes exit (ab)normally at runtime, providing a loose form of structured concurrency

What this will enable us to do is actually spawn each topology component as a dedicated supervisor, with the traditional component task as a significant child in that supervisor, and then give the component a handle to its supervisor so it can spawn whatever other child processes it requires. In doing so, we can ensure all of those descendant child processes are shutdown when the component task itself exits (ab)normally.. but that we can track their association (at a high level) to the component that spawned them, rather than really having no good idea where some arbitrary async task originated from.

Most of the code in this PR is tests: we've altered some of the restart/shutdown behavior not only to support dynamic child processes, but also to properly match the behavior exhibited by Erlang/OTP's supervisor.erl.

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

A lot of new and existing unit tests.

References

DADP-2

[tobz]

• chore(core): unify how we define identifiers for subsystems/components #2029
• chore(core): integrate component tasks via supervision trees #1942
• enhancement(core): add support for dynamically added child processes in Supervisor #1874 👈 (View in Graphite)
• enhancement(core): add configurable per-child restart types for supervisors #1873
• chore(core): clean up a bunch of code smells around supervisable worker implementations #1860
• chore(common): unify multiple shutdown synchronization types #1856
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: f4b7e76 · Comparison: bf9ca0f · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.56 MiB (baseline) vs 40.64 MiB (comparison)
Size Change: +84.15 KiB (+0.20%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
saluki_core::runtime::worker_state	+36.29 KiB	28
hashbrown	+21.00 KiB	100
core	+18.91 KiB	1390
&mut serde_json	+18.24 KiB	12
serde_json	-16.73 KiB	46
tokio	+15.39 KiB	345
anon.a9030c24ed0837a8b572f3c7e56188a8.26.llvm.9410433398480454732	-13.81 KiB	1
anon.6b6ec8798021cdb8f478ee5f74cd462d.10.llvm.12763542860581416599	+13.72 KiB	1
alloc	-12.01 KiB	181
comfy_table	+11.26 KiB	3
chrono	-10.74 KiB	5
saluki_core::runtime::supervisor	-9.70 KiB	143
indexmap	+9.51 KiB	20
anon.da384e72d3643b966148975fcb09a882.6.llvm.13854863512543145371	+9.47 KiB	1
anon.9890a1e0c5f52a6438e911161778df39.22.llvm.7770142864409893306	-9.29 KiB	1
figment	-6.07 KiB	92
tokio_util	-5.49 KiB	20
resource_accounting::groups::Tracked	+5.00 KiB	3
saluki_core::topology::interconnect	-4.87 KiB	9
tower_layer	+4.53 KiB	8

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.7% +42.0Ki  +0.4% +20.1Ki    [6948 Others]
  [NEW] +39.5Ki  [NEW] +39.3Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::ha18bd0afdd86cc2f
  [NEW] +23.7Ki  [NEW] +23.5Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::hda904fa00026b1b1
  [NEW] +21.9Ki  [NEW] +21.8Ki    saluki_core::topology::built::ComponentInterconnects::generate_interconnects::he8cf03e299fdf0bb
  [NEW] +20.6Ki  [NEW] +20.5Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h3dfbaa77715cd8c5
  [NEW] +19.2Ki  [NEW] +19.0Ki    saluki_core::runtime::supervisor::Supervisor::run_inner::_{{closure}}::h7a491cfc16abf6af
  [NEW] +13.7Ki  [NEW]     +82    anon.6b6ec8798021cdb8f478ee5f74cd462d.10.llvm.12763542860581416599
  [NEW] +13.2Ki  [NEW] +13.1Ki    saluki_core::runtime::worker_state::WorkerState::shutdown_workers::_{{closure}}::h9858876562e27ffd
  [NEW] +12.1Ki  [NEW] +11.9Ki    _<resource_accounting::groups::Tracked<Inner> as core::future::future::Future>::poll::h71464a8eacb6e87b
  [NEW] +12.0Ki  [NEW] +11.9Ki    saluki_core::runtime::worker_state::WorkerState::shutdown_workers::_{{closure}}::h3648a616d86e5c62
 +23e2% +11.3Ki +27e2% +11.3Ki    comfy_table::cell::Cell::new::h4b1c39f0a1c85c30
  [NEW] +9.47Ki  [NEW]     +81    anon.da384e72d3643b966148975fcb09a882.6.llvm.13854863512543145371
  [DEL] -9.29Ki  [DEL]     -81    anon.9890a1e0c5f52a6438e911161778df39.22.llvm.7770142864409893306
  [DEL] -9.76Ki  [DEL] -9.65Ki    saluki_core::runtime::supervisor::Supervisor::spawn_child::h0b35b738e03994c4
  [DEL] -10.1Ki  [DEL] -9.94Ki    saluki_core::runtime::supervisor::Supervisor::run_inner::_{{closure}}::h5d92fde1359f3734
  [DEL] -11.5Ki  [DEL] -11.3Ki    _<chrono::format::formatting::DelayedFormat<I> as core::fmt::Display>::fmt::h3c0a1ed65abdbe21
  [DEL] -13.8Ki  [DEL]     -82    anon.a9030c24ed0837a8b572f3c7e56188a8.26.llvm.9410433398480454732
 -88.1% -15.0Ki -88.6% -15.0Ki    saluki_core::topology::built::ComponentInterconnects::from_graph::he08b03929664d596
  [DEL] -20.2Ki  [DEL] -20.1Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::h1e511e3c61e4e2ae
  [DEL] -25.1Ki  [DEL] -24.9Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h49a3ec78d95db62f
  [DEL] -39.8Ki  [DEL] -39.6Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h4d95df7d046da704
  +0.2% +84.1Ki  +0.2% +61.9Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 7466e14f-cf02-4f2d-8920-fb3e57d8c950
Baseline: f4b7e765 · Comparison: bf9ca0f0 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_heavy	memory	⚪ +0.19	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.15	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.04	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.38	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.43	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 131 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 42.9 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 65.4 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 191 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 28.5 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[Copilot]

Pull request overview

This PR enhances saluki-core’s runtime supervision system to support dynamic child processes and bring restart/shutdown behavior closer to Erlang/OTP’s supervisor.erl, enabling patterns like supervising per-connection tasks under a long-lived supervisor.

Changes:

• Added SupervisorHandle / ChildId APIs for spawning workers dynamically while a supervisor is running.
• Introduced ShutdownMode (ordered vs concurrent) and AutoShutdown for significant-child-driven supervisor shutdown.
• Refactored worker bookkeeping into a dedicated WorkerState module and expanded unit tests to cover the new semantics.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

File	Description
lib/saluki-core/src/runtime/worker_state.rs	New worker/task bookkeeping module, including ordered vs concurrent shutdown behavior.
lib/saluki-core/src/runtime/supervisor.rs	Adds dynamic spawn handle/channel, significant child auto-shutdown, shutdown mode, and extensive tests; refactors supervision loop to use WorkerState.
lib/saluki-core/src/runtime/mod.rs	Re-exports new runtime API types and wires in the new worker_state module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 91233f322d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

[webern]

My visual reading of it isn't worth much, I more-or-less understand what's going on. Gave it an opus max pass with no bug findings as well.