fix(otlp): align trace metric attributes and aggregation - #9685
Conversation
Fixes several OTLP trace-metrics (traces.span.sdk.metrics.duration) attribute gaps found in a cross-tracer audit against SEMCON-1093: - datadog.process_tags now emits one array-valued resource attribute (mirroring the legacy v0.6/stats ProcessTags shape) instead of flattening each tag into its own datadog.<key> attribute. - datadog.is_trace_root is now emitted per data point, gated the same way as the other datadog.* attributes. - span.kind is canonicalized to the OTel Span Metrics Connector's SPAN_KIND_* uppercase convention instead of being passed through lowercase. - status.code is now a required string attribute (STATUS_CODE_OK/STATUS_CODE_ERROR) on every data point, replacing the previous OTLP-trace-style int enum that was only set on errors. additional_metric_tags and peer_tags remain unimplemented on this path; both are flagged with a one-line TODO pointing at the corresponding gap (or its absence) in the legacy v0.6/stats exporter.
mabdinur
added
the
AI Generated
Overall package sizeSelf size: 8.2 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.72 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: a849759 | Docs | Datadog PR Page | Give us feedback! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #9685 +/- ##
==========================================
- Coverage 98.55% 98.53% -0.03%
==========================================
Files 966 975 +9
Lines 139420 144148 +4728
Branches 12543 13255 +712
==========================================
+ Hits 137407 142031 +4624
- Misses 2013 2117 +104 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
service.name was omitted from OTLP trace-metrics data points whenever a span's service matched the default/global service, mirroring the status.code fix earlier in this branch: required attributes must be unconditional, not skipped as an optimization.
BenchmarksBenchmark execution time: 2026-08-14 14:03:45 Comparing candidate commit a849759 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 2320 metrics, 38 unstable metrics.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b9775eee23
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85f040093a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3065873246
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f8e8c00fe3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Reverting the change that introduced this overhead. It was introduced by a bug fix for the existing stats where to ensure metrics for spans with local/integration level service names were not combined the metrics for spans with the global service name. This fix can be moved out to it's own PR and optimized separately. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2931d99061
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Native stats without OTLP regressed from a 1.26s to 1.43s benchmark median because service-boundary state was tracked for every span. Removing that tracking restores the master baseline; the behavior will move to an OTLP-gated follow-up.
BridgeAR
left a comment
BridgeAR
left a comment
There was a problem hiding this comment.
I just had a brief look and can not yet determine full correctness
BridgeAR
left a comment
BridgeAR
left a comment
There was a problem hiding this comment.
I do not know the domain, so it is hard to say it works as expected without that.
Code wise, I have one performance suggestion which should reduce the overhead quite a bit :)
Serialize base attributes once per aggregation group and append the top-level and status dimensions for each distribution key. Key-generation microbenchmark, two fresh runs with five-trial medians: 570.6ms to 126.9ms and 569.9ms to 127.3ms.
| const attributes = { | ||
| ...baseAttributes, | ||
| 'datadog.span.top_level': topLevel, | ||
| 'status.code': statusCode, | ||
| } |
There was a problem hiding this comment.
These attributes are only needed for the later transform call which actually just reads the values, maps over them, builds an array with objects and builds special values. We already know the type for statusCode and topLevel, so we know the value shape and we can already calculate the array immediately. That would be even less work and we do not copy any objects here and need less iterations.
Kind of
const transformedAttributes = this.transformAttributes(attributes)
transformedAttributes.push(
{ key: 'datadog.span.top_level', value: { boolValue: topLevel } },
{ key: 'status.code', value: { intValue: statusCode }
)Build the known top-level and status OTLP values directly instead of copying and re-transforming the base attributes.
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
This pull request was merged directly. |
ddb64b7 to
5069eb6
Compare
5069eb6 to
a849759
Compare
* fix(otlp): align trace-metrics attributes with the RFC attribute spec Fixes several OTLP trace-metrics (traces.span.sdk.metrics.duration) attribute gaps found in a cross-tracer audit against SEMCON-1093: - datadog.process_tags now emits one array-valued resource attribute (mirroring the legacy v0.6/stats ProcessTags shape) instead of flattening each tag into its own datadog.<key> attribute. - datadog.is_trace_root is now emitted per data point, gated the same way as the other datadog.* attributes. - span.kind is canonicalized to the OTel Span Metrics Connector's SPAN_KIND_* uppercase convention instead of being passed through lowercase. - status.code is now a required string attribute (STATUS_CODE_OK/STATUS_CODE_ERROR) on every data point, replacing the previous OTLP-trace-style int enum that was only set on errors. additional_metric_tags and peer_tags remain unimplemented on this path; both are flagged with a one-line TODO pointing at the corresponding gap (or its absence) in the legacy v0.6/stats exporter. * fix(otlp): always emit service.name on trace-metrics data points service.name was omitted from OTLP trace-metrics data points whenever a span's service matched the default/global service, mirroring the status.code fix earlier in this branch: required attributes must be unconditional, not skipped as an optimization. * fix(otlp): coalesce trace metrics by exported attributes * fix(otlp): finalize trace metrics semantics * fix(otlp): retain semantic attributes in OTel mode * fix(otlp): defer trace-root attribute detection * test(otlp): reserve only datadog attributes * fix(otlp): omit unknown trace-root attribute * fix(otlp): always emit trace metric attributes * fix(otlp): guard span kind mapping * test(otlp): cover unspecified span kind fallback * fix(otlp): include service-entry metrics and tracer tags * fix(otlp): harden tracer tags and service-entry tracking * fix(otlp): retain services across partial flushes * fix(otlp): weakly retain cached span services * perf(otlp): reduce trace metric processing overhead * perf(otlp): remove service tracking from stats hot path Native stats without OTLP regressed from a 1.26s to 1.43s benchmark median because service-boundary state was tracked for every span. Removing that tracking restores the master baseline; the behavior will move to an OTLP-gated follow-up. * chore(otlp): keep span processor out of core changes * refactor(otlp): rely on initialized resource inputs * refactor(stats): pass trace-root flag directly * perf(otlp): reuse span metric attribute key Serialize base attributes once per aggregation group and append the top-level and status dimensions for each distribution key. Key-generation microbenchmark, two fresh runs with five-trial medians: 570.6ms to 126.9ms and 569.9ms to 127.3ms. * perf(otlp): avoid redundant attribute transforms Build the known top-level and status OTLP values directly instead of copying and re-transforming the base attributes.
* fix(otlp): align trace-metrics attributes with the RFC attribute spec Fixes several OTLP trace-metrics (traces.span.sdk.metrics.duration) attribute gaps found in a cross-tracer audit against SEMCON-1093: - datadog.process_tags now emits one array-valued resource attribute (mirroring the legacy v0.6/stats ProcessTags shape) instead of flattening each tag into its own datadog.<key> attribute. - datadog.is_trace_root is now emitted per data point, gated the same way as the other datadog.* attributes. - span.kind is canonicalized to the OTel Span Metrics Connector's SPAN_KIND_* uppercase convention instead of being passed through lowercase. - status.code is now a required string attribute (STATUS_CODE_OK/STATUS_CODE_ERROR) on every data point, replacing the previous OTLP-trace-style int enum that was only set on errors. additional_metric_tags and peer_tags remain unimplemented on this path; both are flagged with a one-line TODO pointing at the corresponding gap (or its absence) in the legacy v0.6/stats exporter. * fix(otlp): always emit service.name on trace-metrics data points service.name was omitted from OTLP trace-metrics data points whenever a span's service matched the default/global service, mirroring the status.code fix earlier in this branch: required attributes must be unconditional, not skipped as an optimization. * fix(otlp): coalesce trace metrics by exported attributes * fix(otlp): finalize trace metrics semantics * fix(otlp): retain semantic attributes in OTel mode * fix(otlp): defer trace-root attribute detection * test(otlp): reserve only datadog attributes * fix(otlp): omit unknown trace-root attribute * fix(otlp): always emit trace metric attributes * fix(otlp): guard span kind mapping * test(otlp): cover unspecified span kind fallback * fix(otlp): include service-entry metrics and tracer tags * fix(otlp): harden tracer tags and service-entry tracking * fix(otlp): retain services across partial flushes * fix(otlp): weakly retain cached span services * perf(otlp): reduce trace metric processing overhead * perf(otlp): remove service tracking from stats hot path Native stats without OTLP regressed from a 1.26s to 1.43s benchmark median because service-boundary state was tracked for every span. Removing that tracking restores the master baseline; the behavior will move to an OTLP-gated follow-up. * chore(otlp): keep span processor out of core changes * refactor(otlp): rely on initialized resource inputs * refactor(stats): pass trace-root flag directly * perf(otlp): reuse span metric attribute key Serialize base attributes once per aggregation group and append the top-level and status dimensions for each distribution key. Key-generation microbenchmark, two fresh runs with five-trial medians: 570.6ms to 126.9ms and 569.9ms to 127.3ms. * perf(otlp): avoid redundant attribute transforms Build the known top-level and status OTLP values directly instead of copying and re-transforming the base attributes.
2 participants
What does this PR do?
Aligns traces.span.sdk.metrics.duration with the cross-tracer OTLP trace-metrics contract
Motivation
This implements the contract defined in DataDog/system-tests#7363 and refined in DataDog/system-tests#7466 without adding work to the native-stats path when OTLP export is disabled
Additional Notes
Service-boundary discovery is isolated in draft #9774. DD_TRACE_STATS_ADDITIONAL_TAGS and peer tags remain out of scope. Focused unit tests, npm run lint, the native-stats benchmark, and OTLP system tests pass