Skip to content

chore(ci): drop DogStatsD-disable workaround from OTLP tests - #1907

Merged
tobz merged 1 commit into
mainfrom
jszwedko/otlp-tests-remove-dogstatsd-workaround
Jun 23, 2026
Merged

chore(ci): drop DogStatsD-disable workaround from OTLP tests#1907
tobz merged 1 commit into
mainfrom
jszwedko/otlp-tests-remove-dogstatsd-workaround

Conversation

@jszwedko

Copy link
Copy Markdown
Collaborator

Summary

The OTLP correctness tests carried DD_DATA_PLANE_DOGSTATSD_ENABLED=false (added in #1570) purely to work around pre-7.80 misalignment between ADP and Core Agent behavior (#1567) — without it, ADP's DogStatsD pipeline perturbed these OTLP-only baseline-vs-comparison runs. Now that we're on Agent 7.80 that misalignment is gone, so this removes the override (and its TODO) from all six OTLP cases, letting them run with ADP's default pipeline configuration as intended. This resolves the cleanup tracked in #1577.

Closes #1577

Test plan

  • Ran all six affected cases locally (docker runtime, Agent 7.80 baseline) with the override removed — all pass: otlp-metrics, otlp-traces, otlp-traces-ets, otlp-traces-probabilistic, otlp-traces-ottl-filtering, otlp-traces-ottl-transform.

🤖 Generated with Claude Code

#1570 set DD_DATA_PLANE_DOGSTATSD_ENABLED=false in the OTLP correctness
tests to work around pre-7.80 misalignment with Core Agent behavior
(#1567). Now that we run Agent 7.80, ADP's DogStatsD pipeline no longer
perturbs these OTLP-only comparisons, so the override and its TODO are
removed from all six OTLP cases.

Closes #1577

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jszwedko
jszwedko requested a review from a team as a code owner June 23, 2026 02:22
@dd-octo-sts dd-octo-sts Bot added the area/test All things testing: unit/integration, correctness, SMP regression, etc. label Jun 23, 2026
@datadog-datadog-prod-us1-2

Copy link
Copy Markdown
Contributor

Pipelines

Fix all issues with BitsAI

⚠️ Warnings

🚦 1 Pipeline job failed

Semantic PR Title Check | Check For Semantic PR Title   View in Datadog   GitHub Actions

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0b90e57 | Docs | Datadog PR Page | Give us feedback!

@jszwedko jszwedko changed the title test(correctness): drop DogStatsD-disable workaround from OTLP tests chore(ci): drop DogStatsD-disable workaround from OTLP tests Jun 23, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b90e57899

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines 17 to 18
- DD_DATA_PLANE_STANDALONE_MODE=true
- DD_DATA_PLANE_OTLP_ENABLED=true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep DogStatsD disabled while in standalone mode

With these OTLP comparison runs still setting DD_DATA_PLANE_STANDALONE_MODE=true, ADP does not bootstrap the remote-agent config stream that would let Agent 7.80 supply its resolved data_plane.dogstatsd.enabled decision; in this path DataPlaneDogStatsDConfiguration falls back to its default of true. Removing the explicit DD_DATA_PLANE_DOGSTATSD_ENABLED=false therefore re-enables ADP's DogStatsD pipeline in all six OTLP-only cases, so any DogStatsD/internal traffic can again perturb the baseline-vs-comparison results despite the tests intending to validate only OTLP behavior.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct that ADP will own the dogstatsd socket now, but this is actually fine. The Core Agent will read DD_DATA_PLANE_ENABLED, see that is enabled, and that DD_DATA_PLANE_DOGSTATSD_ENABLED isn't set to false so it'll not start dogstatsd (and thus the ports won't conflict).

There could be an argument here to keep dogstatsd disabled in ADP for these specific OTLP tests, but given that the default is that it is enabled, I think it is better to run the OTLP tests with in enabled.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me.

@pr-commenter

pr-commenter Bot commented Jun 23, 2026

Copy link
Copy Markdown

Binary Size Analysis (Agent Data Plane)

Baseline: 2a95db2 · Comparison: 0b90e57 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 40.28 MiB (baseline) vs 40.28 MiB (comparison)
Size Change: +0 B (+0.00%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
anon.c332a4a573d0a9a1bf7c71b0a1f57672.1.llvm.13446777115463930978 +130 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.1.llvm.11552159992296369999 -130 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.4.llvm.13446777115463930978 +115 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.4.llvm.11552159992296369999 -115 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.3.llvm.13446777115463930978 +109 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.3.llvm.11552159992296369999 -109 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.0.llvm.13446777115463930978 +97 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.0.llvm.11552159992296369999 -97 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.2.llvm.13446777115463930978 +95 B 1
anon.c332a4a573d0a9a1bf7c71b0a1f57672.2.llvm.11552159992296369999 -95 B 1
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  [NEW]    +130  [NEW]     +40    anon.c332a4a573d0a9a1bf7c71b0a1f57672.1.llvm.13446777115463930978
  [NEW]    +115  [NEW]     +25    anon.c332a4a573d0a9a1bf7c71b0a1f57672.4.llvm.13446777115463930978
  [NEW]    +109  [NEW]     +19    anon.c332a4a573d0a9a1bf7c71b0a1f57672.3.llvm.13446777115463930978
  [NEW]     +97  [NEW]      +7    anon.c332a4a573d0a9a1bf7c71b0a1f57672.0.llvm.13446777115463930978
  [NEW]     +95  [NEW]      +5    anon.c332a4a573d0a9a1bf7c71b0a1f57672.2.llvm.13446777115463930978
  [DEL]     -95  [DEL]      -5    anon.c332a4a573d0a9a1bf7c71b0a1f57672.2.llvm.11552159992296369999
  [DEL]     -97  [DEL]      -7    anon.c332a4a573d0a9a1bf7c71b0a1f57672.0.llvm.11552159992296369999
  [DEL]    -109  [DEL]     -19    anon.c332a4a573d0a9a1bf7c71b0a1f57672.3.llvm.11552159992296369999
  [DEL]    -115  [DEL]     -25    anon.c332a4a573d0a9a1bf7c71b0a1f57672.4.llvm.11552159992296369999
  [DEL]    -130  [DEL]     -40    anon.c332a4a573d0a9a1bf7c71b0a1f57672.1.llvm.11552159992296369999
  [ = ]       0  [ = ]       0    TOTAL

@pr-commenter

pr-commenter Bot commented Jun 23, 2026

Copy link
Copy Markdown

Regression Detector (Agent Data Plane)

Run ID: 4deffeb8-2ab4-4d6f-a4dd-370e76d41b3a
Baseline: 2a95db2e · Comparison: 0b90e578 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
otlp_ingest_metrics_5mb_memory memory ⚪ +3.19 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic) cpu ⚪ +3.14 metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored) cpu ⚪ +3.04 metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic) cpu ⚪ +2.31 metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored) memory ⚪ +1.50 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.45 metrics profiles logs
dsd_uds_100mb_3k_contexts_memory memory ⚪ +0.44 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory memory ⚪ +0.22 metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic) cpu ⚪ +0.19 metrics profiles logs
dsd_uds_500mb_3k_contexts_memory memory ⚪ +0.02 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput throughput ⚪ -0.01 metrics profiles logs
otlp_ingest_metrics_5mb_throughput throughput ⚪ -0.01 metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput throughput ⚪ -0.00 metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.00 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ -0.00 metrics profiles logs
otlp_ingest_traces_5mb_throughput throughput ⚪ +0.00 metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput throughput ⚪ +0.01 metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored) throughput ⚪ +0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput throughput ⚪ +0.02 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic) cpu ⚪ -0.02 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput throughput ⚪ +0.07 metrics profiles logs
dsd_uds_10mb_3k_contexts_memory memory ⚪ -0.07 metrics profiles logs
otlp_ingest_traces_5mb_memory memory ⚪ -0.13 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory memory ⚪ -0.16 metrics profiles logs
dsd_uds_1mb_3k_contexts_memory memory ⚪ -0.25 metrics profiles logs
dsd_uds_512kb_3k_contexts_memory memory ⚪ -0.37 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ -0.38 metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput throughput ⚪ +0.49 metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic) cpu ⚪ -0.62 metrics profiles logs
quality_gates_rss_idle memory ⚪ -0.76 metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic) cpu ⚪ -0.81 metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic) cpu ⚪ -0.97 metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic) cpu ⚪ -1.21 metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic) cpu ⚪ -4.00 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 133 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 42.2 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 65.8 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 191 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 28.1 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@webern webern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

Comment on lines 17 to 18
- DD_DATA_PLANE_STANDALONE_MODE=true
- DD_DATA_PLANE_OTLP_ENABLED=true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me.

@tobz
tobz merged commit d2eeb82 into main Jun 23, 2026
87 of 88 checks passed
@tobz
tobz deleted the jszwedko/otlp-tests-remove-dogstatsd-workaround branch June 23, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/test All things testing: unit/integration, correctness, SMP regression, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Revert env var workaround in OTLP correctness tests after Agent v7.80

3 participants