Skip to content

Release prep 1.2.4: avoid DNS for disabled ADP bootstrap - #2058

Closed
jszwedko wants to merge 2 commits into
releases/1.2.xfrom
jszwedko/backport-2041-1.2.x
Closed

Release prep 1.2.4: avoid DNS for disabled ADP bootstrap#2058
jszwedko wants to merge 2 commits into
releases/1.2.xfrom
jszwedko/backport-2041-1.2.x

Conversation

@jszwedko

@jszwedko jszwedko commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Release-prep PR for ADP 1.2.4 on releases/1.2.x. Cherry-pick of #2041 plus a version bump.

Backports the fix that avoids DNS lookups during ADP bootstrap when IPC is disabled, preventing startup failures in environments where DNS resolution isn't available.

Commits

Test plan

  • cargo check passes for the affected crates on the 1.2.x toolchain.

🤖 Generated with Claude Code

aqian01 and others added 2 commits July 9, 2026 12:55
- Gate remote-agent bootstrap on `data_plane.enabled` so default-disabled ADP exits before creating the Core Agent IPC client.
- Add an explicit `HttpsCapableConnectorBuilder::without_dns_resolution()` mode.
- Use no-DNS connector construction for Core Agent IPC, which targets `https://127.0.0.1:<cmd_port>` and does not require hostname resolution.
- Avoid system DNS resolver construction for DNS-free connector transports such as Unix sockets and vsock.

On hosts without nameservers in `/etc/resolv.conf`, ADP could fail during startup while constructing the HTTP connector for Core Agent IPC. This happened before ADP reached the disabled-exit path, creating noisy crash loops even when `data_plane.enabled` was false.

unit tests, integration test.

Co-authored-by: andrew.qian <andrew.qian@datadoghq.com>
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 9, 2026 20:01
@jszwedko
jszwedko requested a review from a team as a code owner July 9, 2026 20:01
@dd-octo-sts dd-octo-sts Bot added area/io General I/O and networking. area/test All things testing: unit/integration, correctness, SMP regression, etc. labels Jul 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8882b51695

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

exposed_ports:
- "58125/udp"

procedure:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use the integration assertion schema

This new case is not loadable by Panoramic: IntegrationConfig requires a top-level assertions list, and assertion entries are tagged with type, while this file uses procedure and nested assertion keys. When make test-integration discovers this directory, deserialization fails with the missing assertions field and the discovery path panics, so the integration suite cannot start while this config is present.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 5be14cda44. The test now uses the required top-level assertions field and type-tagged assertion entries, so Panoramic can discover it.

Comment thread test/integration/cases/adp-ipc-no-dns/config.yaml
let ipc_cert_file_path = config.auth().ipc_cert_file_path();
let client_tls_config = build_ipc_client_ipc_tls_config(ipc_cert_file_path).await?;
let connector_builder = HttpsCapableConnectorBuilder::default();
let connector_builder = HttpsCapableConnectorBuilder::default().without_dns_resolution();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve DNS for hostname IPC endpoints

This disables DNS for every remote-agent IPC connector, but RemoteAgentClientConfiguration still supports agent_ipc_endpoint when cmd_port is absent, including non-loopback URIs such as https://agent.example:5001. In that supported configuration the connector is built with the noop resolver, so hostname endpoints fail before they can connect; only disable DNS when the selected endpoint is a literal IP or when a non-DNS transport such as vsock is used.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 5be14cda44. RemoteAgentClient now preserves DNS resolution for hostname endpoints and disables it only for literal IPv4/IPv6 endpoints; vsock continues to bypass DNS.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Release-prep for ADP 1.2.4 on releases/1.2.x, backporting the “avoid DNS during IPC bootstrap” fix to prevent startup failures on hosts without usable DNS configuration.

Changes:

  • Add a “no DNS resolution” mode to the HTTPS-capable HTTP connector builder and unit-test it.
  • Use the no-DNS connector for Core Agent IPC client construction, and add an integration test case for empty /etc/resolv.conf.
  • Update container CA-certs install behavior and bump ADP version to 1.2.4.

Reviewed changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
test/integration/cases/adp-ipc-no-dns/empty-resolv-conf.sh Adds init script to blank /etc/resolv.conf in the integration container.
test/integration/cases/adp-ipc-no-dns/config.yaml Adds an integration test intended to validate ADP startup without DNS.
lib/saluki-io/src/net/client/http/conn.rs Adds without_dns_resolution() to build connectors without constructing a system DNS resolver.
lib/datadog-agent/commons/src/ipc/client/mod.rs Switches IPC client to use the no-DNS connector builder.
docker/scripts/agent-data-plane/app/00-install-ca-certs.sh Adds a script intended to install CA certs only when missing.
docker/Dockerfile.proxy-dumper Removes ca-certificates version pin and adjusts CA-certs install comment.
Cargo.lock Bumps agent-data-plane package version to 1.2.4.
bin/agent-data-plane/Cargo.toml Bumps ADP crate version to 1.2.4.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 25 to 27
RUN test -d /usr/local/share/ca-certificates || apt-get update && \
apt-get install -y --no-install-recommends ca-certificates=20240203 && \
apt-get install -y --no-install-recommends ca-certificates && \
apt-get clean

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in fe6fc2e1c7. The condition now checks /usr/share/ca-certificates and groups the update, install, and cleanup commands so they run only when the directory is absent.

Comment on lines 59 to 63
let auth_interceptor = BearerAuthInterceptor::from_file(&config.auth().auth_token_file_path()).await?;
let ipc_cert_file_path = config.auth().ipc_cert_file_path();
let client_tls_config = build_ipc_client_ipc_tls_config(ipc_cert_file_path).await?;
let connector_builder = HttpsCapableConnectorBuilder::default();
let connector_builder = HttpsCapableConnectorBuilder::default().without_dns_resolution();
#[cfg(target_os = "linux")]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 5be14cda44. DNS resolution is retained for hostname-based agent_ipc_endpoint values and disabled only for literal IPv4/IPv6 endpoints.

/// DNS, such as Unix sockets, vsock, or literal-IP TCP endpoints. Hostname-based TCP
/// destinations will fail to resolve when this is enabled.
///
/// Defaults to enabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 5be14cda44. The documentation now states explicitly that DNS resolution is enabled by default.

Comment on lines +1 to +5
#!/usr/bin/env sh
#
# Ensures CA certificates are present in the final image.
#
# We only install them if they're missing. In CI, the application base image already ships CA

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in fe6fc2e1c7. The unused CA certificate installation script was removed.

Comment on lines +38 to +51
procedure:
- parallel:
- assertion: process_stable_for
duration: 10s
- assertion: log_contains
pattern: "Topology healthy"
timeout: 60s
- assertion: log_not_contains
pattern: "Failed to load system DNS configuration when creating DNS resolver for HTTP client"
during: 10s
- assertion: log_not_contains
pattern: "panic|PANIC"
regex: true
during: 10s

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 5be14cda44. The test now uses assertions and type, matching the Panoramic integration-test schema.

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Jul 9, 2026

Copy link
Copy Markdown

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 6 Pipeline jobs failed

DataDog/saluki | run-benchmarks-adp   View in Datadog   GitLab

DataDog/saluki | test-integration   View in Datadog   GitLab

DataDog/saluki | test-integration-macos-amd64   View in Datadog   GitLab

View all 6 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8882b51 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 9, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: b02c083 · Comparison: 8882b51 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 39.85 MiB (baseline) vs 40.03 MiB (comparison)
Size Change: +182.88 KiB (+0.45%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
figment +87.45 KiB 666
prost -31.49 KiB 482
otlp_protos::otlp_include::opentelemetry +28.43 KiB 191
[sections] +21.70 KiB 7
alloc +20.79 KiB 3501
anon.16459471181944c387c15a5548954026.678.llvm.8738394446259426161 +15.41 KiB 1
anon.c6eff494e958aee50571b85e1a4e41ac.2.llvm.7836766266218528871 -15.32 KiB 1
tonic +14.49 KiB 492
serde_with -12.64 KiB 58
&mut serde_json -12.64 KiB 102
core +9.74 KiB 16175
saluki_common::cache::CacheBuilder<K,V,W,H> +9.51 KiB 6
saluki_components::common::datadog -9.11 KiB 509
agent_data_plane::internal::env +8.72 KiB 199
saluki_components::sources::otlp -7.94 KiB 228
http_body_util -7.77 KiB 255
anon.37b88fe6cb66e39459b6e77ed9b89783.16.llvm.14261231893276215459 +7.77 KiB 1
anon.455e4055ccc158f74b85672ca09e1f2b.260.llvm.221889396750405353 -7.76 KiB 1
anon.058104e7c5e17815cc397c51159e7bb6.17.llvm.5917434062816465002 +7.55 KiB 1
anon.54c1a0044bdc70049609d77194c621d0.1.llvm.2461160726068242817 -7.55 KiB 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.0%  +240Ki  +1.1%  +212Ki    [57756 Others]
  [NEW] +54.4Ki  [NEW] +54.2Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h98629f489dff8344
  [NEW] +44.9Ki  [NEW] +44.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h46f9ae25c2b65de5
  [NEW] +35.5Ki  [NEW] +35.3Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h1e4166e7041e8dcc
  [NEW] +35.3Ki  [NEW] +35.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h93ba9413ffa13406
  [NEW] +30.2Ki  [NEW] +30.1Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hc36da897c8d98729
  [NEW] +28.6Ki  [NEW] +28.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h4f9c1792147b335b
  [NEW] +25.7Ki  [NEW] +25.6Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::he2fe0994b5553485
  [NEW] +25.4Ki  [NEW] +25.2Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h6c61695a2c59aa11
  [NEW] +24.1Ki  [NEW] +24.0Ki    saluki_env::workload::collectors::containerd::NamespaceWatcher::build_initial_metadata_operations::_{{closure}}::h67966dc2d2934319
  [DEL] -24.1Ki  [DEL] -24.0Ki    saluki_env::workload::collectors::containerd::NamespaceWatcher::build_initial_metadata_operations::_{{closure}}::h30ec2069d647fa21
  [DEL] -24.2Ki  [DEL] -24.0Ki    core::ptr::drop_in_place<agent_data_plane::cli::run::handle_run_command::{{closure}}>::h1dd4120ec3ec6d5c
  [DEL] -24.6Ki  [DEL] -24.4Ki    agent_data_plane::internal::remote_agent::run_remote_agent_registration_loop::_{{closure}}::h6ae2b4ba5ed1f5ab
  [DEL] -25.9Ki  [DEL] -25.8Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h499b3cdd6d6beb82
  [DEL] -28.5Ki  [DEL] -28.4Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::haefb58c0f02234a8
  [DEL] -30.1Ki  [DEL] -30.0Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h11e8854c933a8098
  [DEL] -31.6Ki  [DEL] -31.5Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h4e7da8d1ac8d0e9b
  [DEL] -33.5Ki  [DEL] -33.3Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h192ba0b55f6674d3
  [DEL] -35.3Ki  [DEL] -35.1Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::hab94aa8347dcd327
  [DEL] -49.8Ki  [DEL] -49.7Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h2f426cd320078741
  [DEL] -54.3Ki  [DEL] -54.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h737255a90400e679
  +0.4%  +182Ki  +0.5%  +155Ki    TOTAL

@pr-commenter

pr-commenter Bot commented Jul 9, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 626a862a-be04-480c-899f-dcaaabd06daa
Baseline: b02c0834 · Comparison: 8882b516 · diff

Optimization Goals: ❌ 1 regression detected

experiment goal Δ mean % links
otlp_ingest_metrics_5mb_memory memory 🔴 +5.63 metrics profiles logs
Fine details of change detection per experiment (34)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
dsd_uds_1mb_3k_contexts_cpu (erratic) cpu ⚪ +10.84 metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic) cpu ⚪ +8.01 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic) cpu ⚪ +2.51 metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored) memory ⚪ +2.02 metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored) cpu ⚪ +1.38 metrics profiles logs
otlp_ingest_traces_5mb_throughput throughput ⚪ -0.96 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic) cpu ⚪ +0.85 metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic) cpu ⚪ +0.52 metrics profiles logs
dsd_uds_10mb_3k_contexts_memory memory ⚪ +0.47 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ +0.29 metrics profiles logs
otlp_ingest_traces_5mb_memory memory ⚪ +0.13 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.09 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ +0.06 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory memory ⚪ +0.06 metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
otlp_ingest_metrics_5mb_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_100mb_3k_contexts_memory memory ⚪ -0.00 metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored) throughput ⚪ +0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput throughput ⚪ +0.01 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput throughput ⚪ +0.05 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory memory ⚪ -0.07 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput throughput ⚪ +0.08 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.16 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.29 metrics profiles logs
dsd_uds_512kb_3k_contexts_memory memory ⚪ -0.34 metrics profiles logs
dsd_uds_500mb_3k_contexts_memory memory ⚪ -0.66 metrics profiles logs
dsd_uds_1mb_3k_contexts_memory memory ⚪ -0.89 metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic) cpu ⚪ -1.04 metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic) cpu ⚪ -2.46 metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput throughput ⚪ +2.75 metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic) cpu ⚪ -3.16 metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic) cpu 🟢 -12.03 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 135 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 42.1 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 64.4 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 191 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 27.9 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 10, 2026
## Summary

Fixes issues found while reviewing the ADP backport in #2058.

- Keeps DNS enabled for hostname IPC endpoints.
- Disables DNS only for literal-IP and vsock endpoints.
- Fixes the proxy-dumper CA certificate condition.

## Validation

- `make check-fmt`
- `cargo nextest run -p datadog-agent-commons ipc::client::tests`
- `target/debug/panoramic list -d test/integration/cases --runtime linux`

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
@jszwedko jszwedko closed this Jul 10, 2026
gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jul 10, 2026
… backport) (#2080)

## Summary

Backports #2060 onto the ADP 1.2.4 backport in #2058.

- Keeps DNS enabled for hostname IPC endpoints.
- Disables DNS only for literal-IP and vsock endpoints.
- Fixes the 1.2.x integration test schema.
- Fixes the proxy-dumper CA certificate condition and removes the unused certificate script.

## Validation

Unit tests and Panoramic discovery.

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
jszwedko pushed a commit that referenced this pull request Jul 10, 2026
… backport) (#2080)

## Summary

Backports #2060 onto the ADP 1.2.4 backport in #2058.

- Keeps DNS enabled for hostname IPC endpoints.
- Disables DNS only for literal-IP and vsock endpoints.
- Fixes the 1.2.x integration test schema.
- Fixes the proxy-dumper CA certificate condition and removes the unused certificate script.

## Validation

Unit tests and Panoramic discovery.

Co-authored-by: toby.lawrence <toby.lawrence@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/io General I/O and networking. area/test All things testing: unit/integration, correctness, SMP regression, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants