fix(config): permissively coerce scalar config leaves

[webern]

Human Summary

In #2317 @aqian01 detected a flaw in the typed config deserialization mechanism. The ported, permissive, Viper type coercions that the Agent uses were only being applied to environment variables. This means that when we parsed a config file ourselves, we would not have applied those permissive coercions. This is now fixed so that we more completely mirror Agent config parsing behavior when migrating to typed config.

AI Summary

The Datadog Agent accepts configuration values through cast, so a leaf's declared type determines which alternate spellings it accepts. dogstatsd_port: "8125" and a schema-string leaf written as a YAML boolean (use_v3_api.series.enabled: true, read with GetString) are both valid Agent configuration. ADP's generated model instead deserialized each leaf strictly according to its schema type, causing startup translation to fail or rejecting the entire runtime configuration snapshot.

This change:

• Adds serde deserializers for boolean, integer, float, and string schema scalar types.
• Adds a build-time permissivize pass that applies the appropriate deserializer to every generated scalar leaf based on its generated Rust type.
• Makes env_decode use the same parsers, aligning file, environment, and configuration-stream inputs.
• Fails code generation for unrecognized leaf types, so new schema types require an explicit coercion decision.
• Retains hard errors for values that cannot be converted, rather than accepting cast's zero values.
• Removes the now-unnecessary input_shape: string_or_integer metadata and related types.

Field types remain unchanged; only deserialization behavior is permissive. Numeric strings are accepted in decimal form only. The generated model diff adds deserializer attributes, and the change covers the full generated scalar model rather than individual keys.

Change Type

• Bug fix

How did you test this PR?

• Unit tests cover accepted and rejected spellings for each scalar type.
• A schema-driven test exercises every scalar leaf with a castable value and verifies the coerced value through a serialization round-trip.
• Translation-gate tests cover a boolean on the V3 series mode string leaf and a quoted dogstatsd_port.
• make build-schema-overlay, cargo check --workspace --tests, and cargo clippy --workspace --tests pass; config crate tests pass.

References

• Related: fix(config): migrate endpoints to typed config for provenance awareness #2317

---

Note: #2317 merged into the wrong base

I merged #2317 into m/confra-cast (this PR) instead of main. This PR now carries both changes. The #2317 description follows verbatim.

---

Merged from #2317: fix(config): migrate endpoints to typed config for provenance awareness

Human Summary

This migrates all components affected by #1965 (we were unable to detect explicitly set to default vs default config values) to typed config to pick up the fix provided in #2279.

After implementation I ran a clean-room Opus audit with this prompt:

please do a clean-room audit of this commit f4ae503e807b398ca55ee3f6275f554889cff645

Our goal is to correct a previous incorrect behavior in which we were not sensitive to whether a configuration value was set by the Agent as a default or whether the customer explicitly set the value (and happened to set it to the default value).

Other than that specific intentional bug fix, no other behavioral changes are intended.

Validate that only the desired behavioral change has occurred and that configuration defaults are in-tact.

Read AGENTS.md and /config-system

It only found two actionable items, which I fixed. One was that we were previously rejecting empty API keys, so I retained that behavior. The other was that an MRF endpoint consisting only of whitespace could be treated as a valid setting. Also fixed. All defaults stood up to scrutiny.

Edit: @aqian01 found a flaw that went undetected which became #2325.

Review Guide

I tried breaking this up, which is why there are a bunch of subissues being closed all at once, but it did not divide very well. As such, the diff a bit large:

Focus on these files:

• lib/saluki-components/*
• bin/agent-data-plane/src/cli/run.rs
• lib/agent-data-plane-config*

Deprioritize generated code and inventory churn in:

• lib/datadog-agent/*

AI Summary

Migrate Datadog endpoint, retry, proxy, forwarder, metrics encoder, MRF, and Cluster Agent configuration consumers from raw GenericConfiguration deserialization to resolved typed configuration.

Primary endpoint resolution now happens once in the typed configuration layer:

• A default-sourced dd_url no longer shadows site.
• An explicitly configured dd_url, including the schema-default URL, remains an override.
• MRF and Cluster Agent destination overrides cannot be overwritten by global endpoint settings.
• Retry queue size precedence uses configuration provenance, so explicit zero values are preserved.
• Raw configuration access remains only for live API-key refresh, secrets retry behavior, and the run_path compatibility fallback whose schema default is still an unresolved placeholder.

The schema inventories, generated registries, smoke-test metadata, and component tests are updated for the typed consumers.

Change Type

• Non-functional (chore, refactoring, docs)

How did you test this PR?

• unit tests updated and created throughout
• integration test modified to test this behavior. this was proven to have caught the original bug (red -> green)

References

• Closes: Investigate whether dd_url's schema default makes site unreachable in ADP. #1965
• Closes: Migrate Datadog metrics configuration to typed config for #1965 #2310
• Closes: Migrate Datadog forwarder configuration to typed config for #1965 #2312
• Closes: Migrate shared forwarder configuration to typed config for #1965 #2313
• Closes: Make endpoint configuration provenance-aware for #1965 #2314
• Closes: Make retry configuration provenance-aware for #1965 #2315
• Closes: Migrate cluster-agent forwarder configuration to typed config for #1965 #2316
• Related to: chore(config): preserve provenance in typed configuration #2279

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 6825bb3 · Comparison: fa1f541 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 41.65 MiB (baseline) vs 41.21 MiB (comparison)
Size Change: -457.53 KiB (-1.07%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
figment	-159.13 KiB	230
tracing	-142.54 KiB	96
saluki_common::resource_tracking::groups	+139.08 KiB	41
core	-119.77 KiB	9935
saluki_components::common::datadog	-58.80 KiB	543
agent_data_plane_config_system::saluki_env_overlay::PathRecorder	+55.18 KiB	21
agent_data_plane_config_system::saluki_only::_	-42.91 KiB	9
[sections]	-27.30 KiB	9
serde	-21.45 KiB	43
anon.a0ca6a6a6723caf97f9e22a220a3e9ae.917.llvm.12682091795628431370	+18.14 KiB	1
anon.b55517ae7242af04bd1e7e3ada3e3db4.915.llvm.200922871829932699	-18.13 KiB	1
anon.091bbad26cc3924e9c77bf8d4e018d37.460.llvm.16286174339815826116	-17.53 KiB	1
saluki_components::transforms::dogstatsd_mapper	-17.26 KiB	20
anon.b04cc4d39dfb6a1b14a3e51699c12cc3.656.llvm.4117878566411149896	+17.09 KiB	1
axum	-16.24 KiB	402
alloc	+15.27 KiB	1753
saluki_components::sources::dogstatsd	-15.24 KiB	416
serde_core	+13.08 KiB	807
&mut serde_json	-12.83 KiB	113
tonic	+12.67 KiB	422

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +59.9Ki  [NEW] +59.7Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::hd9fc277b29cf0b9d
  [NEW] +48.1Ki  [NEW] +48.0Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::haafc62f62f4023d0
  [NEW] +39.6Ki  [NEW] +39.5Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h23953cd93c271a64
  [NEW] +39.4Ki  [NEW] +39.2Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h580bce97a0c152b9
  [NEW] +38.5Ki  [NEW] +38.3Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h1f6e2f36e5c43c76
  [NEW] +35.8Ki  [NEW] +35.5Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h6a606a7931ae90c3
  [NEW] +33.3Ki  [NEW] +33.1Ki    agent_data_plane::internal::env::workload::RemoteAgentWorkloadProvider::from_configuration::_{{closure}}::h396a15ba33ef7d0b
  [NEW] +31.4Ki  [NEW] +31.2Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::h3146495241b183c8
  [NEW] +31.2Ki  [NEW] +31.0Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::h8e6a26bba5766166
  [NEW] +28.5Ki  [NEW] +28.4Ki    agent_data_plane::dogstatsd_contexts::artifact::for_each_record::h39b2832027986acc
  [DEL] -29.2Ki  [DEL] -29.0Ki    _<figment::value::de::ConfiguredValueDe<I> as serde_core::de::Deserializer>::deserialize_struct::hd8309826b77c3c8b
  [DEL] -31.3Ki  [DEL] -31.2Ki    agent_data_plane::cli::dogstatsd::run_dogstatsd_command::_{{closure}}::hdb76b647c6fe8e15
  [DEL] -31.5Ki  [DEL] -31.3Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hd4786a9338182043
  [DEL] -35.0Ki  [DEL] -34.7Ki    _<agent_data_plane_config_system::saluki_only::_::<impl serde_core::de::Deserialize for agent_data_plane_config_system::saluki_only::SalukiOnly>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h792c51a7e4c37d3d
  [DEL] -36.9Ki  [DEL] -36.7Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hbcbb45ee17ef2879
  [DEL] -38.4Ki  [DEL] -38.3Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h4556528d50f12a07
  [DEL] -40.1Ki  [DEL] -39.9Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h99e6e806e208f4e8
  [DEL] -41.6Ki  [DEL] -41.3Ki    _<saluki_components::common::datadog::config::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::config::ForwarderConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h9fd5615a048f22e7
  [DEL] -42.3Ki  [DEL] -42.1Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hb5fa11377218b0bc
  [DEL] -60.4Ki  [DEL] -60.2Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h5f9e60794ac4b144
  -2.3%  -456Ki  -2.7%  -407Ki    [38992 Others]
  -1.1%  -457Ki  -1.1%  -407Ki    TOTAL

[datadog-prod-us1-5]

The new coercion layer still rejects two forms accepted by the Agent’s pinned cast implementation: floating-point booleans and zero-fraction integer strings. Either form can make ADP’s strict configuration gate fail while the Agent accepts the same setting.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit bcb9c8c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[webern]

• fix(config): migrate endpoints to typed config for provenance awareness #2317
• fix(config): permissively coerce scalar config leaves #2325 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 2b2a7bbc-3752-4749-b087-6c539df937a3
Baseline: 6825bb3d · Comparison: fa1f541b · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.67	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.42	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.15	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ -0.62	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -1.00	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 229 MiB ≤ 250 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 50.6 MiB ≤ 60 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 90.3 MiB ≤ 100 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 381 MiB ≤ 420 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 31.7 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[webern]

What's going on here? Are we storing system.config().shared.metrics_encoding.v3_series_mode.mode as a String instead of a bool? Why? In typed config we should be storing values in system-native types (i.e. as a primitive boolean) not as a config-friendly-string that the system later has to coerce.

Suggested change

	assert_eq!(system.config().shared.metrics_encoding.v3_series_mode.mode, "true");
	assert_eq!(system.config().shared.metrics_encoding.v3_series_mode.mode, true);

[webern]

Turns out this field is not a boolean. It is an enum (defined in schema documentation rather than typed), with the values true, false and datadog_only. So ea9db67 models it as an enum following an established pattern we have for these and documents the pattern in the config-system skill.

[aqian01]

I think the start up path still uses the raw config

Code trace:

1. passes raw map to dogstatsd
2. Uses the raw config
3. Constructed from raw config
4. And the field requires u16

Ran a regression test to confirm

#[tokio::test]
async fn dogstatsd_accepts_quoted_port() {
    let values = json!({ "dogstatsd_port": "8126" });
    let (config, _) = ConfigurationLoader::for_tests(Some(values), None, false).await;
    let config = DogStatsDConfiguration::from_configuration(&config)
        .expect("quoted port should work");

    assert_eq!(config.port, 8126);
}

[webern]

I think the start up path still uses the raw config

Code trace:

1. passes raw map to dogstatsd
2. Uses the raw config
3. Constructed from raw config
4. And the field requires u16

Ran a regression test to confirm

#[tokio::test]
async fn dogstatsd_accepts_quoted_port() {
    let values = json!({ "dogstatsd_port": "8126" });
    let (config, _) = ConfigurationLoader::for_tests(Some(values), None, false).await;
    let config = DogStatsDConfiguration::from_configuration(&config)
        .expect("quoted port should work");

    assert_eq!(config.port, 8126);
}

This is the behavior on main as well, confirmed by running your test there. This PR does not change any raw config path behavior. It makes the typed configuration model coerce scalar leaves the way the Agent does, which will cover dogstatsd_port when migrated. DogStatsDConfiguration still reads raw_map(), so the fix for dogstatsd_port arrives when that component is migrated to typed config.