Skip to content

chore(agent-data-plane): share shutdown signal wait logic - #2323

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 7 commits into
mainfrom
jszwedko/register-shutdown-signal-early
Aug 13, 2026
Merged

chore(agent-data-plane): share shutdown signal wait logic#2323
gh-worker-dd-mergequeue-cf854d[bot] merged 7 commits into
mainfrom
jszwedko/register-shutdown-signal-early

Conversation

@jszwedko

@jszwedko jszwedko commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Human Summary

Refactors the shutdown wait handler into a shared module that can be used where ever we need to wait to ensure we consistently rely on the same signals, cross-platform.

AI Summary

#2322 added SIGTERM/SIGINT handling for graceful shutdown, but the same signal-waiting logic was duplicated between the run command and dogstatsd replay's cancellation setup. An earlier version of this PR tried to close a related gap -- a signal delivered before the supervisor starts falling through to the OS's default disposition -- by registering signal handlers eagerly at process start. Per discussion on #2322, that gap doesn't need graceful handling: an abrupt exit before the supervisor is running is acceptable. This PR instead does a pure refactor, extracting the existing lazy signal-wait logic into a shared saluki_app::util::wait_for_shutdown_signal helper used by both run and dogstatsd replay, removing the duplication and making it reusable by any binary built on the saluki crates, not just agent-data-plane.

Test plan

  • cargo check --workspace, cargo clippy -p agent-data-plane -p saluki-app --no-deps pass.
  • No new automated tests added; this is a non-behavioral refactor of existing signal-handling code.

jszwedko and others added 4 commits August 12, 2026 14:58
SIGTERM is the signal used by systemd, container runtimes, and Kubernetes to
request shutdown, but ADP only listened for SIGINT, so it terminated
immediately without draining the topology or honoring the shutdown timeout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…dows

On Windows, ADP is managed as a subprocess by dd-procmgr, which requests a
graceful stop by sending CTRL_BREAK_EVENT (not CTRL_C_EVENT) via
GenerateConsoleCtrlEvent. The previous fallback only listened for
tokio::signal::ctrl_c(), so this request went unnoticed and ADP would sit
until dd-procmgr's stop timeout expired and force-killed it via its Job
Object, reproducing the same "no graceful shutdown" behavior as SIGTERM
on Unix.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ulary

The new SIGTERM/CTRL_BREAK doc comment on wait_for_shutdown_signal uses
these terms, which Vale's spelling check doesn't recognize, failing
check-docs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signal registration via `tokio::signal::unix::signal`/`ctrl_c` happens
synchronously when called, but the previous code only called it inside the
`async fn` polled at the point we're ready to wait for shutdown -- so a
signal delivered anywhere during startup (e.g. while waiting on the Datadog
Agent's initial configuration) fell through to the OS default disposition
and killed the process immediately, skipping cleanup.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@pr-commenter

pr-commenter Bot commented Aug 12, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 6825bb3 · Comparison: d601d0f · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.65 MiB (baseline) vs 41.53 MiB (comparison)
Size Change: -132.69 KiB (-0.31%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
core -53.34 KiB 1663
[sections] -20.04 KiB 7
anon.091bbad26cc3924e9c77bf8d4e018d37.460.llvm.16286174339815826116 -17.53 KiB 1
anon.24f983bdb5d7a9f41626c5544756a394.887.llvm.10441113739384331192 +17.52 KiB 1
tokio -14.51 KiB 916
agent_data_plane_config::shared::_ +12.79 KiB 17
anon.f6b518799a8b19c7e356431bcdfa99c4.533.llvm.6184102975340891627 +12.29 KiB 1
anon.091bbad26cc3924e9c77bf8d4e018d37.513.llvm.16286174339815826116 -12.29 KiB 1
tracing_subscriber -11.45 KiB 62
agent_data_plane_config::domains::dogstatsd -10.43 KiB 13
anon.24f983bdb5d7a9f41626c5544756a394.180.llvm.10441113739384331192 +9.72 KiB 1
anon.61f4b4fd3b48a953f00eec7e253961a0.298.llvm.6300809537654130731 -9.72 KiB 1
anon.24f983bdb5d7a9f41626c5544756a394.882.llvm.10441113739384331192 +9.16 KiB 1
anon.61f4b4fd3b48a953f00eec7e253961a0.559.llvm.6300809537654130731 -9.15 KiB 1
serde_core +8.68 KiB 97
anon.091bbad26cc3924e9c77bf8d4e018d37.4.llvm.16286174339815826116 -8.58 KiB 1
anon.2c9e2e2154d21bde9e49f2d4896e12f7.12.llvm.2573425551792850089 +8.40 KiB 1
serde_json -7.94 KiB 113
smallvec -7.74 KiB 30
anon.f6b518799a8b19c7e356431bcdfa99c4.392.llvm.6184102975340891627 +6.94 KiB 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +17.5Ki  [NEW]     +76    anon.24f983bdb5d7a9f41626c5544756a394.887.llvm.10441113739384331192
  [NEW] +14.0Ki  [NEW] +13.9Ki    agent_data_plane::state::metrics::rules::compat::get_compat_remappings::h19651ee55cb484c9
  [NEW] +12.3Ki  [NEW]     +81    anon.f6b518799a8b19c7e356431bcdfa99c4.533.llvm.6184102975340891627
  [NEW] +12.0Ki  [NEW] +11.9Ki    agent_data_plane::state::metrics::rules::dogstatsd::get_dogstatsd_remappings::hfb3b72311acf3cad
  [NEW] +9.72Ki  [NEW]     +79    anon.24f983bdb5d7a9f41626c5544756a394.180.llvm.10441113739384331192
  [NEW] +9.16Ki  [NEW]     +91    anon.24f983bdb5d7a9f41626c5544756a394.882.llvm.10441113739384331192
  [NEW] +8.40Ki  [NEW]     +81    anon.2c9e2e2154d21bde9e49f2d4896e12f7.12.llvm.2573425551792850089
  [NEW] +6.94Ki  [NEW]     +77    anon.f6b518799a8b19c7e356431bcdfa99c4.392.llvm.6184102975340891627
 -84.8% -6.73Ki -86.3% -6.73Ki    _<h2::client::Connection<T,B> as core::future::future::Future>::poll::h0f7c777c6a66efe3
  [DEL] -6.86Ki  [DEL]     -77    anon.091bbad26cc3924e9c77bf8d4e018d37.399.llvm.16286174339815826116
  [DEL] -7.08Ki  [DEL] -6.89Ki    serde_json::value::de::_<impl serde_core::de::Deserialize for serde_json::value::Value>::deserialize::h45fdda11a534df1b
  [DEL] -8.27Ki  [DEL] -8.18Ki    serde_core::ser::SerializeMap::serialize_entry::hcd94be53b1549835
  [DEL] -8.58Ki  [DEL]     -81    anon.091bbad26cc3924e9c77bf8d4e018d37.4.llvm.16286174339815826116
  [DEL] -9.15Ki  [DEL]     -91    anon.61f4b4fd3b48a953f00eec7e253961a0.559.llvm.6300809537654130731
  [DEL] -9.72Ki  [DEL]     -79    anon.61f4b4fd3b48a953f00eec7e253961a0.298.llvm.6300809537654130731
  [DEL] -12.3Ki  [DEL]     -81    anon.091bbad26cc3924e9c77bf8d4e018d37.513.llvm.16286174339815826116
  -1.1% -13.3Ki  -1.1% -13.3Ki    [section .gcc_except_table]
  [DEL] -14.8Ki  [DEL] -14.7Ki    agent_data_plane::state::metrics::rules::get_compat_remappings::hc7620de93b7e14a3
  [DEL] -17.5Ki  [DEL]     -76    anon.091bbad26cc3924e9c77bf8d4e018d37.460.llvm.16286174339815826116
 -98.1% -24.9Ki -98.5% -24.9Ki    agent_data_plane::state::metrics::rules::get_datadog_agent_remappings::h4ff388cc81845209
  -0.8% -83.5Ki  -0.6% -51.1Ki    [10166 Others]
  -0.3%  -132Ki  -0.3% -100.0Ki    TOTAL

@pr-commenter

pr-commenter Bot commented Aug 12, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 4c63ab98-e19b-4926-a3ca-3e81fe8f5261
Baseline: 6825bb3d · Comparison: d601d0f9 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
quality_gates_rss_dsd_heavy memory ⚪ +0.36 metrics profiles logs
quality_gates_rss_idle memory ⚪ +0.04 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ -0.00 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.11 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ -0.43 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 229 MiB ≤ 250 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 51.3 MiB ≤ 60 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 91.7 MiB ≤ 100 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 389 MiB ≤ 420 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 31.9 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

Comment thread bin/agent-data-plane/src/cli/run.rs Outdated
Comment thread bin/agent-data-plane/src/cli/run.rs Outdated
Base automatically changed from jszwedko/handle-sigterm-adp to main August 13, 2026 14:10
jszwedko and others added 2 commits August 13, 2026 10:35
…tdown-signal-early

# Conflicts:
#	bin/agent-data-plane/src/cli/run.rs
…`run` and `dogstatsd replay`

Per discussion on #2322, a signal received before the supervisor starts doesn't need graceful handling, so this drops the earlier eager pre-registration approach and instead extracts the existing lazy signal-wait logic into a shared helper reused by both the `run` and `dogstatsd replay` commands, removing the duplication.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jszwedko jszwedko changed the title fix(agent-data-plane): install shutdown signal handlers before startup chore(agent-data-plane): share shutdown signal wait logic Aug 13, 2026
@jszwedko
jszwedko marked this pull request as ready for review August 13, 2026 18:01
@jszwedko
jszwedko requested a review from a team as a code owner August 13, 2026 18:01

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The shared helper preserves the run command’s platform-specific signal branches and replay’s signal-to-cancellation transition. Its replay import and caller remain Linux-gated, so the extraction does not introduce a cross-platform compilation or shutdown-flow regression.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 52ee2de · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@jszwedko
jszwedko requested a review from tobz August 13, 2026 18:36
…uki-app

Moves the shared shutdown-signal-wait logic into `saluki_app::util` so it's reusable by any binary built on the saluki crates, not just agent-data-plane.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@dd-octo-sts dd-octo-sts Bot added the area/observability Internal observability of ADP and Saluki. label Aug 13, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit e760adf into main Aug 13, 2026
100 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the jszwedko/register-shutdown-signal-early branch August 13, 2026 20:43
dd-octo-sts Bot pushed a commit that referenced this pull request Aug 13, 2026
## Human Summary

Refactors the shutdown wait handler into a shared module that can be used where ever we need to wait to ensure we consistently rely on the same signals, cross-platform.

## AI Summary

#2322 added SIGTERM/SIGINT handling for graceful shutdown, but the same signal-waiting logic was duplicated between the `run` command and `dogstatsd replay`'s cancellation setup. An earlier version of this PR tried to close a related gap -- a signal delivered before the supervisor starts falling through to the OS's default disposition -- by registering signal handlers eagerly at process start. Per [discussion on #2322](#2322 (comment)), that gap doesn't need graceful handling: an abrupt exit before the supervisor is running is acceptable. This PR instead does a pure refactor, extracting the existing lazy signal-wait logic into a shared `saluki_app::util::wait_for_shutdown_signal` helper used by both `run` and `dogstatsd replay`, removing the duplication and making it reusable by any binary built on the saluki crates, not just agent-data-plane.

## Test plan

- [x] `cargo check --workspace`, `cargo clippy -p agent-data-plane -p saluki-app --no-deps` pass.
- No new automated tests added; this is a non-behavioral refactor of existing signal-handling code.

Co-authored-by: jesse.szwedko <jesse.szwedko@datadoghq.com> e760adf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/observability Internal observability of ADP and Saluki. mergequeue-status: done

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants