chore(config): support environment variables in the config system

[webern]

Human Summary

This PR should unblock the typed config project. Issues arose from trying to deserialize typed config from GenericConfiguration because Figment was doing a sort of heuristic coercion of environment variables without knowledge of their destination types. I believe this is actually lossy, but either way, it was easier to remove Figment from the deserialization and translation pathway (since we have the Datadog schema) than it was to try to deserialize from Figment's model of the config map.

In this PR we hand the initial file read over to the typed config system which deserializes it and applies environment variable precedence to it (configurable) based on the information we have in the schema and a port of the environment variable handling algorithms as found in the Agent.

Ownership of the config stream is given to the new typed config system, and GenericConfiguration and the typed system operate independently with it. First the typed system merges what it receives into a copy its own serde_json::Value map, then it updates typed configuration and sends the ConfigUpdate on to the GenericConfiguration map which behaves as it always has, using Figment, etc.

If an error occurs during typed config deserialization and translation, that error will be logged, but the ConfigUpdate will be sent on to GenericConfiguration anyway in order to preserve current system behavior. The thinking behind this is that during a transition to typed configuration, we want to leave things running the same way the have been, and we will add configuration tests to make sure we have no error logs.

We want to gain confidence that we aren't seeing errors during deserialization or translation and quickly switch over to blocking the ConfigUpdate from reaching GenericConfiguration in the presence of an error so that the two views of config cannot drift from one another. The ultimate goal, really, is going to be to migrate everything over to typed config as quickly as possible so that two views of the config do not even exist in the system.

This PR is separate from #2094 so that this PR can land without touching agent-data-plane. The integration with agent-data-plane takes place in the next PR, but looking at that PR will be helpful in terms of understanding how this integrates with run.rs and system startup.

AI Summary

Support environment variables whose flat, underscore-joined names correspond to nested typed configuration paths.

• Add a generated overlay for Datadog configuration keys before typed deserialization.
• Discover and overlay Saluki-only nested paths from the typed source model.
• Support disabled, fallback, and override overlay modes while preserving the Agent configuration stream's precedence in fallback mode.
• Preserve the Agent's whitespace-separated handling for string-list environment values.

The overlays are applied per source before deserialization, so relocating keys for one source does not affect the other.

Change Type

• Non-functional (chore, refactoring, docs)

How did you test this PR?

• Conducted an extensive clean-room audit of with independent, no-context model reviews; found and fixed numeric byte-size compatibility and full-path duration-codegen collisions.
• All the usual local gates with AI-added unit tests for new functionality
• CI

References

Stacked on #1986.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: f750b4c · Comparison: 0f451da · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 39.54 MiB (baseline) vs 39.55 MiB (comparison)
Size Change: +6.03 KiB (+0.01%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
saluki_components::common::datadog	+50.23 KiB	578
serde	-38.27 KiB	88
figment	-23.17 KiB	611
anon.b4a080b658dde80e49dc9c0e9caef940.430.llvm.13566929057820997797	-16.29 KiB	1
anon.a77611365f0a648e74874ca5f00a0131.430.llvm.16394682052481819650	+16.21 KiB	1
otlp_protos::otlp_include::opentelemetry	+14.30 KiB	70
piecemeal	-13.25 KiB	42
core	+12.56 KiB	9039
anon.1c0bbc15a89def2f3945aa257f6125e6.359.llvm.2476128395837609489	+11.67 KiB	1
anon.165201492e5477eec85eff0926ad7758.359.llvm.1031680609193366154	-11.66 KiB	1
datadog_protos::trace_piecemeal_include::datadog	+11.25 KiB	23
rmp	+9.71 KiB	38
anon.a77611365f0a648e74874ca5f00a0131.472.llvm.16394682052481819650	+9.54 KiB	1
anon.b4a080b658dde80e49dc9c0e9caef940.472.llvm.13566929057820997797	-9.45 KiB	1
prost	-9.19 KiB	233
&mut rmp_serde	-9.01 KiB	8
anon.a77611365f0a648e74874ca5f00a0131.399.llvm.16394682052481819650	+8.90 KiB	1
anon.b4a080b658dde80e49dc9c0e9caef940.399.llvm.13566929057820997797	-8.80 KiB	1
anon.1c0bbc15a89def2f3945aa257f6125e6.0.llvm.2476128395837609489	+8.04 KiB	1
anon.165201492e5477eec85eff0926ad7758.0.llvm.1031680609193366154	-8.04 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW] +59.3Ki  [NEW] +59.1Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h7857932b4e694248
  +0.4% +57.4Ki  +0.5% +55.1Ki    [32575 Others]
  [NEW] +43.5Ki  [NEW] +43.3Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h33bc899b6767875d
  [NEW] +41.6Ki  [NEW] +41.5Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h1dedd9d452cdab9a
  [NEW] +36.8Ki  [NEW] +36.5Ki    _<saluki_components::common::datadog::obfuscation::_::<impl serde_core::de::Deserialize for saluki_components::common::datadog::obfuscation::ObfuscationConfig>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h4a8b71a428367082
  [NEW] +35.5Ki  [NEW] +35.3Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h81c7868ee673e106
  [NEW] +31.6Ki  [NEW] +31.5Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::h623f33ccdbfe0105
  [NEW] +30.6Ki  [NEW] +30.5Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h7179f9ced4694e6b
  [NEW] +30.3Ki  [NEW] +30.1Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::h5d2e2f478b16df0c
  [NEW] +29.7Ki  [NEW] +29.5Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::h684cad2b8b8f7918
  [DEL] -27.7Ki  [DEL] -27.6Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hac2a7bcde4780228
  [DEL] -28.3Ki  [DEL] -28.1Ki    saluki_components::sources::dogstatsd::drive_stream::_{{closure}}::h104376214dcc290b
  [DEL] -29.4Ki  [DEL] -29.2Ki    _<saluki_components::sources::dogstatsd::_::<impl serde_core::de::Deserialize for saluki_components::sources::dogstatsd::DogStatsDConfiguration>::deserialize::__Visitor as serde_core::de::Visitor>::visit_map::h883e03a06cc41da6
  [DEL] -30.3Ki  [DEL] -30.1Ki    datadog_agent_commons::ipc::client::RemoteAgentClient::from_client_configuration::_{{closure}}::_{{closure}}::_{{closure}}::hf9f87d53da52cfdf
  [DEL] -30.6Ki  [DEL] -30.5Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::h09e30375dbbfc7a5
  [DEL] -31.1Ki  [DEL] -30.9Ki    _<saluki_components::transforms::aggregate::Aggregate as saluki_core::components::transforms::Transform>::run::_{{closure}}::hd26f038711b53cd8
  [DEL] -32.8Ki  [DEL] -32.7Ki    saluki_components::sources::otlp::metrics::translator::OtlpMetricsTranslator::translate_metrics::ha95697fb5f5066a9
  [DEL] -35.6Ki  [DEL] -35.4Ki    _<saluki_components::transforms::apm_stats::ApmStats as saluki_core::components::transforms::Transform>::run::_{{closure}}::h426bca0ec788e1a2
  [DEL] -41.6Ki  [DEL] -41.5Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h269a10ac30b70c85
  [DEL] -43.6Ki  [DEL] -43.4Ki    _<saluki_components::forwarders::otlp::OtlpForwarder as saluki_core::components::forwarders::Forwarder>::run::_{{closure}}::h29a76a4ba92d0553
  [DEL] -59.1Ki  [DEL] -59.0Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h993d5c57ebf263bd
  +0.0% +6.03Ki  +0.0% +3.96Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: bac8d1a5-ca40-4fe7-af7b-de7940c5b8db
Baseline: f750b4cc · Comparison: 0f451dab · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (5)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
quality_gates_rss_dsd_low	memory	⚪ +0.40	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.24	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +0.22	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ -0.06	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ -0.11	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 133 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 44.1 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 65.9 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 193 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 30.1 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[webern]

• chore(config): cutover adp bin components to typed config system #1995
• chore(config): cutover saluki-components to the new typed configuration system #1987
• chore(config): build the first components from the typed configuration model #1979
• chore(config): integrate and test the config system #2094
• chore(config): support environment variables in the config system #2093 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 90794cc38c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-prod-us1-3]

The csv_comma_separated decoder silently truncates input when unexpected non-comma characters follow a closing quote in a quoted field (e.g., "a"rest,"b" returns ["a"]), diverging from Go's LazyQuotes behavior which would include the characters in the field. This bug is latent: no generated env key currently uses CsvCommaSeparated (0 entries in the generated table), so production is not affected today. It will become reachable when apm_config.ignore_resources (the sole YAML field with env_parser: csv_comma_separated) is added to the typed Rust model.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 90794cc · What is Autotest? · Any feedback? Reach out in #autotest}