[DSM] Test: verify DSM produce checkpoints with ASB 7.18.0 by robcarlan-datadog · Pull Request #8429 · DataDog/dd-trace-dotnet

robcarlan-datadog · 2026-04-08T20:05:35Z

Summary

Bump the DSM Azure Service Bus test sample from 7.14.0 to 7.18.0
Enable the DSM integration test in CI (was previously SkipInCI)
Pre-create DSM entities in the emulator config since the admin API is unavailable
No tracer code changes — this PR only tests whether DSM produce checkpoints work with ASB 7.18.0

Motivation

We suspect DSM produce checkpoints may be broken for Azure Service Bus SDK 7.18.x+. The DSM test was previously hardcoded to 7.14.0 and skipped in CI, so there was no coverage.

If the ValidateSpanTags test fails (fewer than 9 spans with pathway.hash), it confirms the produce-side DSM checkpoint is broken and we need a fix.

Test plan

Watch CI — expect DataStreamsMonitoringAzureServiceBusTests to either pass (DSM works) or fail (confirms the bug)

🤖 Generated with Claude Code

Bump the DSM test sample to ASB 7.18.0 and enable the test in CI to verify whether DSM produce checkpoints are broken for this SDK version. - Bump Azure.Messaging.ServiceBus from 7.14.0 to 7.18.0 in DSM sample - Replace SkipInCI with RequiresDockerDependency/DockerGroup traits - Pre-create DSM entities in emulator config (admin API not available) - Skip admin API calls in sample app when using the emulator Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

dd-trace-dotnet-ci-bot · 2026-04-08T20:52:32Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (8429) and master.

✅ No regressions detected - check the details below

Full Metrics Comparison

FakeDbCommand

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	78.35 ± (78.23 - 78.64) ms	77.97 ± (77.83 - 78.31) ms	-0.5%	✅
.NET Framework 4.8 - Bailout
duration	83.24 ± (83.08 - 83.48) ms	82.78 ± (82.65 - 83.11) ms	-0.6%	✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1130.25 ± (1129.60 - 1135.61) ms	1123.09 ± (1125.49 - 1133.52) ms	-0.6%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	24.14 ± (24.08 - 24.20) ms	23.52 ± (23.45 - 23.58) ms	-2.6%	✅
process.time_to_main_ms	92.96 ± (92.73 - 93.19) ms	90.58 ± (90.35 - 90.80) ms	-2.6%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.90 ± (10.89 - 10.90) MB	10.92 ± (10.91 - 10.92) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	12 ± (12 - 12)	12 ± (12 - 12)	+0.0%	✅
.NET Core 3.1 - Bailout
process.internal_duration_ms	24.00 ± (23.92 - 24.07) ms	23.66 ± (23.60 - 23.72) ms	-1.4%	✅
process.time_to_main_ms	93.45 ± (93.20 - 93.70) ms	93.09 ± (92.86 - 93.32) ms	-0.4%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.93 ± (10.93 - 10.93) MB	10.95 ± (10.95 - 10.95) MB	+0.2%	✅⬆️
runtime.dotnet.threads.count	13 ± (13 - 13)	13 ± (13 - 13)	+0.0%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	238.96 ± (237.87 - 240.05) ms	236.42 ± (235.18 - 237.67) ms	-1.1%	✅
process.time_to_main_ms	555.89 ± (554.41 - 557.36) ms	548.87 ± (547.47 - 550.28) ms	-1.3%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	48.25 ± (48.22 - 48.28) MB	48.41 ± (48.38 - 48.43) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 29)	+0.7%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	22.55 ± (22.49 - 22.61) ms	22.30 ± (22.23 - 22.37) ms	-1.1%	✅
process.time_to_main_ms	80.39 ± (80.12 - 80.66) ms	79.09 ± (78.85 - 79.33) ms	-1.6%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.60 ± (10.60 - 10.60) MB	10.64 ± (10.63 - 10.64) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 6 - Bailout
process.internal_duration_ms	22.43 ± (22.38 - 22.49) ms	22.06 ± (22.00 - 22.11) ms	-1.7%	✅
process.time_to_main_ms	81.68 ± (81.45 - 81.91) ms	79.68 ± (79.43 - 79.94) ms	-2.4%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	10.72 ± (10.72 - 10.72) MB	10.75 ± (10.74 - 10.75) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	384.87 ± (382.68 - 387.06) ms	384.77 ± (382.51 - 387.04) ms	-0.0%	✅
process.time_to_main_ms	557.43 ± (556.25 - 558.62) ms	547.86 ± (546.80 - 548.92) ms	-1.7%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	49.81 ± (49.78 - 49.84) MB	49.97 ± (49.94 - 50.00) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	28 ± (28 - 28)	28 ± (28 - 28)	+0.0%	✅⬆️
.NET 8 - Baseline
process.internal_duration_ms	20.88 ± (20.81 - 20.94) ms	20.36 ± (20.31 - 20.41) ms	-2.5%	✅
process.time_to_main_ms	79.42 ± (79.21 - 79.63) ms	77.37 ± (77.17 - 77.56) ms	-2.6%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.68 ± (7.67 - 7.68) MB	7.66 ± (7.66 - 7.67) MB	-0.2%	✅
runtime.dotnet.threads.count	10 ± (10 - 10)	10 ± (10 - 10)	+0.0%	✅
.NET 8 - Bailout
process.internal_duration_ms	20.60 ± (20.55 - 20.66) ms	20.55 ± (20.48 - 20.61) ms	-0.3%	✅
process.time_to_main_ms	79.90 ± (79.71 - 80.10) ms	79.31 ± (79.09 - 79.52) ms	-0.7%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	7.72 ± (7.71 - 7.73) MB	7.72 ± (7.71 - 7.73) MB	-0.0%	✅
runtime.dotnet.threads.count	11 ± (11 - 11)	11 ± (11 - 11)	+0.0%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	312.71 ± (308.51 - 316.90) ms	307.39 ± (304.92 - 309.86) ms	-1.7%	✅
process.time_to_main_ms	517.81 ± (516.79 - 518.83) ms	511.67 ± (510.76 - 512.58) ms	-1.2%	✅
runtime.dotnet.exceptions.count	0 ± (0 - 0)	0 ± (0 - 0)	+0.0%	✅
runtime.dotnet.mem.committed	37.19 ± (37.16 - 37.22) MB	37.15 ± (37.12 - 37.18) MB	-0.1%	✅
runtime.dotnet.threads.count	27 ± (27 - 27)	27 ± (27 - 27)	+0.0%	✅⬆️

HttpMessageHandler

Metric	Master (Mean ± 95% CI)	Current (Mean ± 95% CI)	Change	Status
.NET Framework 4.8 - Baseline
duration	213.28 ± (212.70 - 213.77) ms	210.65 ± (210.54 - 211.39) ms	-1.2%	✅
.NET Framework 4.8 - Bailout
duration	216.65 ± (216.21 - 217.04) ms	215.77 ± (215.31 - 216.35) ms	-0.4%	✅
.NET Framework 4.8 - CallTarget+Inlining+NGEN
duration	1234.72 ± (1233.51 - 1241.59) ms	1231.66 ± (1232.72 - 1241.80) ms	-0.2%	✅
.NET Core 3.1 - Baseline
process.internal_duration_ms	209.20 ± (208.73 - 209.68) ms	206.59 ± (206.14 - 207.04) ms	-1.2%	✅
process.time_to_main_ms	91.16 ± (90.83 - 91.50) ms	90.00 ± (89.66 - 90.35) ms	-1.3%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	15.92 ± (15.89 - 15.94) MB	15.92 ± (15.90 - 15.93) MB	-0.0%	✅
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	+0.2%	✅⬆️
.NET Core 3.1 - Bailout
process.internal_duration_ms	210.24 ± (209.69 - 210.78) ms	206.86 ± (206.36 - 207.36) ms	-1.6%	✅
process.time_to_main_ms	93.21 ± (92.95 - 93.48) ms	91.89 ± (91.60 - 92.18) ms	-1.4%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	15.98 ± (15.96 - 15.99) MB	16.03 ± (16.01 - 16.05) MB	+0.4%	✅⬆️
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (21 - 21)	-0.2%	✅
.NET Core 3.1 - CallTarget+Inlining+NGEN
process.internal_duration_ms	424.58 ± (423.03 - 426.13) ms	417.48 ± (416.18 - 418.78) ms	-1.7%	✅
process.time_to_main_ms	556.93 ± (555.52 - 558.34) ms	550.35 ± (548.95 - 551.76) ms	-1.2%	✅
runtime.dotnet.exceptions.count	3 ± (3 - 3)	3 ± (3 - 3)	+0.0%	✅
runtime.dotnet.mem.committed	59.17 ± (59.11 - 59.22) MB	59.34 ± (59.30 - 59.37) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	+0.1%	✅⬆️
.NET 6 - Baseline
process.internal_duration_ms	214.80 ± (214.28 - 215.33) ms	213.00 ± (212.50 - 213.51) ms	-0.8%	✅
process.time_to_main_ms	79.63 ± (79.39 - 79.86) ms	78.83 ± (78.58 - 79.07) ms	-1.0%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.18 ± (16.15 - 16.20) MB	16.25 ± (16.23 - 16.27) MB	+0.5%	✅⬆️
runtime.dotnet.threads.count	20 ± (19 - 20)	20 ± (19 - 20)	-0.3%	✅
.NET 6 - Bailout
process.internal_duration_ms	214.14 ± (213.55 - 214.72) ms	211.10 ± (210.54 - 211.66) ms	-1.4%	✅
process.time_to_main_ms	80.87 ± (80.63 - 81.12) ms	79.99 ± (79.71 - 80.28) ms	-1.1%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	16.19 ± (16.17 - 16.21) MB	16.29 ± (16.26 - 16.31) MB	+0.6%	✅⬆️
runtime.dotnet.threads.count	21 ± (21 - 21)	21 ± (20 - 21)	-0.3%	✅
.NET 6 - CallTarget+Inlining+NGEN
process.internal_duration_ms	597.93 ± (595.58 - 600.28) ms	598.42 ± (596.08 - 600.76) ms	+0.1%	✅⬆️
process.time_to_main_ms	557.45 ± (556.24 - 558.66) ms	549.44 ± (548.25 - 550.63) ms	-1.4%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	61.38 ± (61.28 - 61.48) MB	61.40 ± (61.28 - 61.51) MB	+0.0%	✅⬆️
runtime.dotnet.threads.count	31 ± (31 - 31)	31 ± (31 - 31)	-0.4%	✅
.NET 8 - Baseline
process.internal_duration_ms	212.68 ± (212.15 - 213.20) ms	211.49 ± (211.00 - 211.99) ms	-0.6%	✅
process.time_to_main_ms	78.30 ± (78.04 - 78.55) ms	78.44 ± (78.15 - 78.74) ms	+0.2%	✅⬆️
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.55 ± (11.53 - 11.57) MB	11.58 ± (11.56 - 11.60) MB	+0.3%	✅⬆️
runtime.dotnet.threads.count	19 ± (19 - 19)	19 ± (19 - 19)	+0.8%	✅⬆️
.NET 8 - Bailout
process.internal_duration_ms	212.92 ± (212.45 - 213.39) ms	211.12 ± (210.60 - 211.63) ms	-0.8%	✅
process.time_to_main_ms	79.76 ± (79.52 - 80.01) ms	79.64 ± (79.40 - 79.89) ms	-0.2%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	11.59 ± (11.57 - 11.60) MB	11.66 ± (11.64 - 11.67) MB	+0.6%	✅⬆️
runtime.dotnet.threads.count	20 ± (20 - 20)	20 ± (20 - 20)	-0.5%	✅
.NET 8 - CallTarget+Inlining+NGEN
process.internal_duration_ms	572.77 ± (564.60 - 580.94) ms	563.45 ± (555.48 - 571.42) ms	-1.6%	✅
process.time_to_main_ms	513.58 ± (512.47 - 514.69) ms	513.23 ± (512.29 - 514.17) ms	-0.1%	✅
runtime.dotnet.exceptions.count	4 ± (4 - 4)	4 ± (4 - 4)	+0.0%	✅
runtime.dotnet.mem.committed	51.22 ± (51.12 - 51.32) MB	51.14 ± (51.04 - 51.23) MB	-0.2%	✅
runtime.dotnet.threads.count	30 ± (30 - 30)	30 ± (30 - 30)	-0.5%	✅

Comparison explanation

Execution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

Duration charts

FakeDbCommand (.NET Framework 4.8)

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (78ms)  : 74, 82
    master - mean (78ms)  : 76, 81

    section Bailout
    This PR (8429) - mean (83ms)  : 79, 86
    master - mean (83ms)  : 81, 86

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (1,130ms)  : 1069, 1190
    master - mean (1,133ms)  : 1088, 1177

FakeDbCommand (.NET Core 3.1)

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (121ms)  : 117, 126
    master - mean (124ms)  : 120, 129

    section Bailout
    This PR (8429) - mean (124ms)  : 119, 129
    master - mean (125ms)  : 121, 129

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (822ms)  : 795, 850
    master - mean (836ms)  : 816, 856

FakeDbCommand (.NET 6)

gantt
    title Execution time (ms) FakeDbCommand (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (108ms)  : 104, 113
    master - mean (110ms)  : 104, 116

    section Bailout
    This PR (8429) - mean (108ms)  : 103, 113
    master - mean (111ms)  : 106, 116

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (963ms)  : 921, 1004
    master - mean (969ms)  : 924, 1013

FakeDbCommand (.NET 8)

gantt
    title Execution time (ms) FakeDbCommand (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (106ms)  : 102, 110
    master - mean (108ms)  : 104, 113

    section Bailout
    This PR (8429) - mean (108ms)  : 105, 111
    master - mean (109ms)  : 105, 112

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (851ms)  : 814, 889
    master - mean (858ms)  : 803, 912

HttpMessageHandler (.NET Framework 4.8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (211ms)  : 207, 215
    master - mean (213ms)  : 207, 219

    section Bailout
    This PR (8429) - mean (216ms)  : 211, 221
    master - mean (217ms)  : 213, 221

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (1,237ms)  : 1167, 1307
    master - mean (1,238ms)  : 1179, 1296

HttpMessageHandler (.NET Core 3.1)

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (306ms)  : 299, 313
    master - mean (310ms)  : 303, 317

    section Bailout
    This PR (8429) - mean (308ms)  : 301, 316
    master - mean (313ms)  : 305, 320

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (1,006ms)  : 969, 1042
    master - mean (1,021ms)  : 989, 1053

HttpMessageHandler (.NET 6)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (301ms)  : 291, 311
    master - mean (304ms)  : 297, 311

    section Bailout
    This PR (8429) - mean (300ms)  : 292, 307
    master - mean (304ms)  : 297, 311

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (1,181ms)  : 1142, 1219
    master - mean (1,187ms)  : 1153, 1221

HttpMessageHandler (.NET 8)

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8)
    dateFormat  x
    axisFormat %Q
    todayMarker off
    section Baseline
    This PR (8429) - mean (301ms)  : 293, 308
    master - mean (301ms)  : 293, 310

    section Bailout
    This PR (8429) - mean (301ms)  : 294, 309
    master - mean (303ms)  : 295, 312

    section CallTarget+Inlining+NGEN
    This PR (8429) - mean (1,107ms)  : 994, 1220
    master - mean (1,120ms)  : 993, 1246

github-actions bot added the area:data-streams-monitoring label Apr 8, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DSM] Test: verify DSM produce checkpoints with ASB 7.18.0#8429

[DSM] Test: verify DSM produce checkpoints with ASB 7.18.0#8429
robcarlan-datadog wants to merge 1 commit intomasterfrom
rob/fix-asb-dsm-produce-checkpoint

robcarlan-datadog commented Apr 8, 2026

Uh oh!

dd-trace-dotnet-ci-bot bot commented Apr 8, 2026

FakeDbCommand

HttpMessageHandler

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

robcarlan-datadog commented Apr 8, 2026

Summary

Motivation

Test plan

Uh oh!

dd-trace-dotnet-ci-bot bot commented Apr 8, 2026

Execution-Time Benchmarks Report ⏱️

FakeDbCommand

HttpMessageHandler

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant