Fix broken parenting when Ocelot API Gateway is used#8128
Fix broken parenting when Ocelot API Gateway is used#8128
Conversation
BenchmarksBenchmark execution time: 2026-03-27 18:49:03 Comparing candidate commit 3a108a5 in PR branch Found 4 performance improvements and 4 performance regressions! Performance is the same for 261 metrics, 19 unstable metrics.
|
Execution-Time Benchmarks Report ⏱️Execution-time results for samples comparing This PR (8128) and master. ✅ No regressions detected - check the details below Full Metrics ComparisonFakeDbCommand
HttpMessageHandler
Comparison explanationExecution-time benchmarks measure the whole time it takes to execute a program, and are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are highlighted in **red**. The following thresholds were used for comparing the execution times:
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard. Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph). Duration chartsFakeDbCommand (.NET Framework 4.8)gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (76ms) : 73, 78
master - mean (73ms) : 70, 76
section Bailout
This PR (8128) - mean (80ms) : 77, 84
master - mean (76ms) : 75, 78
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,109ms) : 1067, 1151
master - mean (1,098ms) : 1039, 1157
FakeDbCommand (.NET Core 3.1)gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (118ms) : 113, 124
master - mean (119ms) : 116, 123
section Bailout
This PR (8128) - mean (119ms) : 114, 124
master - mean (121ms) : 117, 125
section CallTarget+Inlining+NGEN
This PR (8128) - mean (803ms) : 783, 822
master - mean (818ms) : 795, 841
FakeDbCommand (.NET 6)gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (105ms) : 101, 109
master - mean (106ms) : 103, 110
section Bailout
This PR (8128) - mean (106ms) : 103, 108
master - mean (107ms) : 105, 109
section CallTarget+Inlining+NGEN
This PR (8128) - mean (954ms) : 923, 986
master - mean (956ms) : 918, 993
FakeDbCommand (.NET 8)gantt
title Execution time (ms) FakeDbCommand (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (104ms) : 100, 108
master - mean (104ms) : 101, 107
section Bailout
This PR (8128) - mean (105ms) : 102, 109
master - mean (105ms) : 103, 107
section CallTarget+Inlining+NGEN
This PR (8128) - mean (841ms) : 805, 877
master - mean (836ms) : 801, 870
HttpMessageHandler (.NET Framework 4.8)gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (214ms) : 209, 219
master - mean (214ms) : 207, 220
section Bailout
This PR (8128) - mean (219ms) : 213, 225
master - mean (218ms) : 212, 223
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,253ms) : 1188, 1318
master - mean (1,226ms) : 1164, 1287
HttpMessageHandler (.NET Core 3.1)gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (313ms) : 306, 321
master - mean (302ms) : 293, 311
section Bailout
This PR (8128) - mean (314ms) : 307, 322
master - mean (303ms) : 297, 309
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,022ms) : 997, 1046
master - mean (999ms) : 974, 1024
HttpMessageHandler (.NET 6)gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (305ms) : 296, 314
master - mean (293ms) : 285, 301
section Bailout
This PR (8128) - mean (306ms) : 300, 313
master - mean (291ms) : 284, 299
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,193ms) : 1157, 1228
master - mean (1,179ms) : 1140, 1219
HttpMessageHandler (.NET 8)gantt
title Execution time (ms) HttpMessageHandler (.NET 8)
dateFormat x
axisFormat %Q
todayMarker off
section Baseline
This PR (8128) - mean (306ms) : 299, 312
master - mean (298ms) : 290, 306
section Bailout
This PR (8128) - mean (306ms) : 299, 313
master - mean (297ms) : 291, 304
section CallTarget+Inlining+NGEN
This PR (8128) - mean (1,117ms) : 983, 1252
master - mean (1,080ms) : 994, 1166
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ae314c0 to
88228c0
Compare
b9bec75 to
1547cf8
Compare
1547cf8 to
7d03568
Compare
2340187 to
6aaa0a6
Compare
6aaa0a6 to
1ab66fb
Compare
1ab66fb to
334375f
Compare
So just to confirm, the original issue doesn't repro on earlier versions? |
andrewlock
left a comment
There was a problem hiding this comment.
LGTM, just a couple of minor things:
- We generally try to cover the full range of supported TFMs, so we should probably support .NET 6+ as Ocelot does
- Should this be the
HttpMessageHandleror a newOcelotintegration? - Couple of nits (using the more compact aspnetcore setup, using combinatorial data attr etc for future)
| [EditorBrowsable(EditorBrowsableState.Never)] | ||
| public sealed class OcelotMessageInvokerPoolIntegration | ||
| { | ||
| private const string IntegrationName = nameof(Configuration.IntegrationId.HttpMessageHandler); |
There was a problem hiding this comment.
Should this be a new integration, Ocelot, instead of HttpMessageHandler? 🤔 I'm undecided tbh, I get the reasoning.
There was a problem hiding this comment.
Yeah, this initially was Ocelot but Yarp went with HttpMessageHandler so I just forced Claude to follow that
Honestly I'm not sure what is ideal
There was a problem hiding this comment.
I could probably go both ways, but we aren't really "instrumenting" YARP/Ocelot here (YARP uses HttpMessageHandler as well).
I think based on that and that we are essentially attempting to fix re-parenting issues here not calling it Ocelot is the better representation, but I would say that if for some reason someone needs to disable this re-parenting fix I guess they would ideally want the option to do that without having to disable all of the HttpMessageHandler instrumentation.
Maybe a second PR to update both YARP and this to be controlled by either HttpMessageHandler or their integration name itself?
tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OcelotDistributedTracingTests.cs
Outdated
Show resolved
Hide resolved
tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OcelotDistributedTracingTests.cs
Outdated
Show resolved
Hide resolved
tracer/test/Datadog.Trace.ClrProfiler.IntegrationTests/OcelotDistributedTracingTests.cs
Outdated
Show resolved
Hide resolved
...ions/integrations/Samples.Ocelot.DistributedTracing/Samples.Ocelot.DistributedTracing.csproj
Outdated
Show resolved
Hide resolved
tracer/test/test-applications/integrations/Samples.Ocelot.DistributedTracing/Program.cs
Outdated
Show resolved
Hide resolved
5cd33a5 to
566fef8
Compare
Correct I couldn't get it to reproduce in earlier versions |
ce38e3e to
780e797
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
This resolves a reparenting issue when OpenTelemetry.Instrumentation.Http is present and running.
This demonstrates the difference in parenting when we have Ocelot and OpenTelemetry compared to when we have the instrumenation that gets rid of the parenting misconfiguration. Note that in this snapshot there is an "ID_3" referenced as the "ParentId", but that there is no "ID_3" present in the snapshot.
Just doing to this help with reviewers to see what changed / what it should be
Cover Ocelot 23.x which supports net6.0-net8.0, not just net8.0+.
3a108a5 to
ae9d877
Compare
Summary of changes
Add auto-instrumentation for Ocelot API gateway to fix broken distributed traces when Ocelot is used with
DD_TRACE_OTEL_ENABLEDandOpenTelemetry.Instrumentation.Httpis used.Reason for change
When customers use Ocelot v23.0.0+ alongside the
OpenTelemetry.Instrumentation.HttpNuGet package, distributed traces break. Ocelot creates aSocketsHttpHandlerwith a defaultActivityHeadersPropagator(this is normal behavior), which causesDiagnosticsHandlerto be added to the handler chain. When the OpenTelemetry HTTP instrumentation is registered, it subscribes to theDiagnosticSourceevents fromDiagnosticsHandlerand injects its own trace context headers into downstream requests which overwrites thex-datadog-*/ W3Ctraceparentheaders already set by ourHttpClientinstrumentation.Without the OpenTelemetry HTTP instrumentation NuGet installed, the
DiagnosticsHandlerActivity events have no subscriber to inject competing headers, so the issue does not occur.For customers this results in broken parenting for traces/spans.
Implementation details
OcelotMessageInvokerPoolIntegration: InstrumentsOcelot.Requester.MessageInvokerPool.CreateHandlerto setActivityHeadersPropagator = nullon the returnedSocketsHttpHandler, preventingDiagnosticsHandlerfrom being added to the handler chain.[InstrumentMethod]attributes handle the return type change between versions:HttpMessageHandlerSocketsHttpHandlerNote: YARP needs to be updated as well from
CreateNoOutputPropagator()tonullI'll do this in a follow upTest coverage
Samples.Ocelot.DistributedTracingthat has Ocelot configured as a reverse proxy andOpenTelemetry.Instrumentation.Httpinstalled and setup. Note that not havingOpenTelemetry.Instrumentation.Httpcauses it to not re-produce.OcelotDistributedTracingTestswhich follow the same pattern as the YARP testsOther details
Supported Ocelot versions: 23.0.0 to 24.*
Earlier versions do not have the same behavior as
HttpMessageHandlerwas introduced first in 23 and then changed toSocketsHttpHandlerin 24.