fix(config): migrate endpoints to typed config for provenance awareness

[webern]

Human Summary

This migrates all components affected by #1965 (we were unable to detect explicitly set to default vs default config values) to typed config to pick up the fix provided in #2279.

After implementation I ran a clean-room Opus audit with this prompt:

please do a clean-room audit of this commit f4ae503e807b398ca55ee3f6275f554889cff645

Our goal is to correct a previous incorrect behavior in which we were not sensitive to whether a configuration value was set by the Agent as a default or whether the customer explicitly set the value (and happened to set it to the default value).

Other than that specific intentional bug fix, no other behavioral changes are intended.

Validate that only the desired behavioral change has occurred and that configuration defaults are in-tact.

Read AGENTS.md and /config-system

It only found two actionable items, which I fixed. One was that we were previously rejecting empty API keys, so I retained that behavior. The other was that an MRF endpoint consisting only of whitespace could be treated as a valid setting. Also fixed. All defaults stood up to scrutiny.

Edit: @aqian01 found a flaw that went undetected which became #2325.

Review Guide

I tried breaking this up, which is why there are a bunch of subissues being closed all at once, but it did not divide very well. As such, the diff a bit large:

Focus on these files:

• lib/saluki-components/*
• bin/agent-data-plane/src/cli/run.rs
• lib/agent-data-plane-config*

Deprioritize generated code and inventory churn in:

• lib/datadog-agent/*

AI Summary

Migrate Datadog endpoint, retry, proxy, forwarder, metrics encoder, MRF, and Cluster Agent configuration consumers from raw GenericConfiguration deserialization to resolved typed configuration.

Primary endpoint resolution now happens once in the typed configuration layer:

• A default-sourced dd_url no longer shadows site.
• An explicitly configured dd_url, including the schema-default URL, remains an override.
• MRF and Cluster Agent destination overrides cannot be overwritten by global endpoint settings.
• Retry queue size precedence uses configuration provenance, so explicit zero values are preserved.
• Raw configuration access remains only for live API-key refresh, secrets retry behavior, and the run_path compatibility fallback whose schema default is still an unresolved placeholder.

The schema inventories, generated registries, smoke-test metadata, and component tests are updated for the typed consumers.

Change Type

• Non-functional (chore, refactoring, docs)

How did you test this PR?

• unit tests updated and created throughout
• integration test modified to test this behavior. this was proven to have caught the original bug (red -> green)

References

• Closes: Investigate whether dd_url's schema default makes site unreachable in ADP. #1965
• Closes: Migrate Datadog metrics configuration to typed config for #1965 #2310
• Closes: Migrate Datadog forwarder configuration to typed config for #1965 #2312
• Closes: Migrate shared forwarder configuration to typed config for #1965 #2313
• Closes: Make endpoint configuration provenance-aware for #1965 #2314
• Closes: Make retry configuration provenance-aware for #1965 #2315
• Closes: Migrate cluster-agent forwarder configuration to typed config for #1965 #2316
• Related to: chore(config): preserve provenance in typed configuration #2279

[webern]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• fix(config): migrate endpoints to typed config for provenance awareness #2317 👈 (View in Graphite)
• fix(config): permissively coerce scalar config leaves #2325
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: ca31d5f · Comparison: fd640ba · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.48 MiB (baseline) vs 41.32 MiB (comparison)
Size Change: -159.85 KiB (-0.38%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
agent_data_plane_config_system::saluki_env_overlay::PathRecorder	+54.70 KiB	23
core	-45.21 KiB	11957
agent_data_plane_config_system::saluki_only::_	-42.61 KiB	9
serde_core	-41.42 KiB	883
serde	-39.81 KiB	50
tonic	-29.84 KiB	418
saluki_components::common::datadog	-24.91 KiB	536
http_body_util	+23.39 KiB	168
prost	-21.67 KiB	391
figment	-20.75 KiB	340
anon.a0ca6a6a6723caf97f9e22a220a3e9ae.898.llvm.12599596679146302898	+17.71 KiB	1
anon.b55517ae7242af04bd1e7e3ada3e3db4.896.llvm.2388164327066883074	-17.71 KiB	1
anon.77fe6d9fde1247be6ecad764c1853154.512.llvm.7927372867784889850	-17.44 KiB	1
anon.56564439fab9ed60f53510d1ef4a1297.566.llvm.213134945675610472	+17.17 KiB	1
std	+16.50 KiB	411
piecemeal	+14.36 KiB	51
otlp_protos::otlp_include::opentelemetry	-14.24 KiB	188
saluki_common::cache::CacheBuilder<K,V,W,H>	+12.58 KiB	8
&mut serde_json	-12.54 KiB	119
anon.77fe6d9fde1247be6ecad764c1853154.595.llvm.7927372867784889850	-12.38 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +60.3Ki  [NEW] +60.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hd9fc277b29cf0b9d
  [NEW] +45.7Ki  [NEW] +45.6Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::haafc62f62f4023d0
  [NEW] +40.5Ki  [NEW] +40.3Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h580bce97a0c152b9
  [NEW] +38.6Ki  [NEW] +38.5Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h23953cd93c271a64
  [NEW] +36.1Ki  [NEW] +35.9Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h1f6e2f36e5c43c76
  [NEW] +32.9Ki  [NEW] +32.7Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h396a15ba33ef7d0b
  [NEW] +30.5Ki  [NEW] +30.3Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::h3146495241b183c8
  [NEW] +30.5Ki  [NEW] +30.3Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h8e6a26bba5766166
  [NEW] +28.5Ki  [NEW] +28.4Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::h39b2832027986acc
  [NEW] +28.1Ki  [NEW] +27.8Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h6a606a7931ae90c3
  [DEL] -27.6Ki  [DEL] -27.4Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::h7fa333d6ccbb3bdc
  [DEL] -28.4Ki  [DEL] -28.3Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::h51a5676cc38d40ed
  [DEL] -30.8Ki  [DEL] -30.6Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hd4786a9338182043
  [DEL] -31.6Ki  [DEL] -31.5Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::hdb76b647c6fe8e15
  [DEL] -34.8Ki  [DEL] -34.6Ki    _<agent_data_plane_config_system::saluki_only::_::<impl serde_core::de::Deserialize for agent_data_plane_config_system::saluki_only::SalukiOnly>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h792c51a7e4c37d3d
  [DEL] -35.1Ki  [DEL] -34.9Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h99e6e806e208f4e8
  [DEL] -37.7Ki  [DEL] -37.6Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h4556528d50f12a07
  [DEL] -38.1Ki  [DEL] -37.9Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hbcbb45ee17ef2879
  [DEL] -39.5Ki  [DEL] -39.4Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hb5fa11377218b0bc
  [DEL] -59.9Ki  [DEL] -59.8Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h5f9e60794ac4b144
  -0.7%  -167Ki  -0.7%  -127Ki    [42645 Others]
  -0.4%  -159Ki  -0.3%  -120Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 5d8afe64-dd28-404e-89fd-9676f7493eff
Baseline: ca31d5fc · Comparison: fd640ba6 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.21	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.48	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.76	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ -0.78	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.84	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 226 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 50.6 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 91.4 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 382 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.3 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[datadog-datadog-prod-us1-2]

More details

The provenance-based endpoint and retry resolution paths preserve the intended distinctions between defaulted and explicitly configured values, including explicit schema-default URLs and zero queue sizes. Local execution was blocked by the unavailable pinned Rust toolchain and an unfetchable git dependency, so runtime confidence is limited to the surrounding code and existing test design.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit f9aaad6 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[aqian01]

Do we want to remove this? I think this is a behavior change since this causes us to not accept true as a boolean now?

Ran this regression test to confirm

#[test]
fn boolean_use_v3_api_series_enabled_is_normalized() {
    let sources = SourceTree::all_explicit(json!({
        "use_v3_api": {
            "series": {
                "enabled": true
            }
        }
    }));

    let config = translate_strict(&sources)
        .expect("a boolean V3 series mode should translate");

    assert_eq!(
        config.shared.metrics_encoding.v3_series_mode.mode,
        "true"
    );
}

[webern]

This was a very important finding (perhaps, see below). I fixed it in a separate PR that is now a precursor to this one (this one has now been stacked on the fix).

To prove the where it was detected, I added your test in this PR at lib/agent-data-plane-config-system/src/system.rs around line 419.

The flaw may not have been as bad as it seemed at first. I believe the Agent actually sends us remote config post-type-coercion, and the Agent-ported coercion algorithms were already being used for ADP-local environment variables. So I think this would have only surfaced when ADP was parsing a local YAML file itself.

In any event, the fix is in #2325 and your helpful test now passes!

[datadog-datadog-prod-us1-2]

More details

The typed-config cutover preserves the reviewed production paths: endpoint provenance now distinguishes explicit default URLs from Agent defaults, retry-size precedence retains explicit zero values, and MRF/Cluster Agent overrides remain isolated from global routing. No additional behavioral regression was supported by the diff and nearby tests; runtime execution was intentionally not performed.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 532719d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}