perf(native): encode finalized JS spans in batches - #9838
Conversation
… span types + exporter) Introduces the native-spans pipeline as a self-contained subsystem. Spans flow through @DataDog/libdatadog's WASM-backed TraceExporter via a change-buffer protocol instead of being formatted in JS and shipped over the agent's HTTP endpoint. native/index.js — libdatadog pipeline loader. - Lazy require + maybeLoad; exposes `available`, OpCode, WasmSpanState, wasmMemory, plus lazy class re-exports via a loadWithNoop helper that prevents fs-instrumentation recursion during module loading. - Split try/catch: MODULE_NOT_FOUND is silent (expected on platforms without libdatadog); other require errors get log.warn (corrupt package / EACCES on the .node binary / etc.); pipeline.init or setStorage failures get log.error. native/native_spans.js — NativeSpansInterface, the JS-side bridge. - Slot allocator (u32 slot indices, not raw spanIds). - String table with rollback-on-failure ordering: WASM-side insert runs before the JS-side map set, so a thrown insert never leaves the JS map claiming a string is interned at a dangling id. - Change-buffer wire format (header + per-op records). Three queue methods: queueOp, queueCreateSpan (op=13), plus the batch helpers queueBatchMeta (op=15) / queueBatchMetrics (op=16). The class-level JSDoc documents the byte layout. - Detach-safety invariant: every WASM call that can grow memory is followed by #checkDetach() to refresh the cached _cqbView / _cqbBytes views. No entry-time check on queue methods; the inner getStringId loop self-heals before any view writes happen. - Atomic setAgentUrl(): builds the new WasmSpanState before clearing JS-side bookkeeping, so a thrown WasmSpanState constructor leaves the existing state consistent. - flushChangeQueue() and flushSpans() rethrow after resetting JS-side state + refreshing views, so callers get a loud signal instead of a silent half-flushed buffer. - Periodic stats flush registered via globalThis dd-trace beforeExitHandlers with a process.once fallback. native/span.js + native/span_context.js — NativeDatadogSpan and NativeSpanContext. - NativeDatadogSpan extends DatadogSpan, overriding only the methods that need native-storage sync (constructor, _createContext, setTag, _addTags, finish). Baggage, links/events, scope, util.inspect, sanitization helpers, and the rest of the OpenTracing surface are inherited unchanged. - NativeSpanContext extends DatadogSpanContext. Slot-indexed; `_name` setter queues SetName via _syncNameToNative, with a construction- time no-op shadow so the parent constructor's initial name write doesn't double-emit alongside queueCreateSpan. - Batched-sync hot path for _addTags: a plain-object input writes directly to the JS cache + syncToNativeOnly in one pass. Priority short-circuits prioritySampler.sample() once a priority is decided. - #serializeSpanLinks / #serializeSpanEvents apply MAX_META_VALUE_LENGTH truncation matching the JS exporter path; oversized payloads would otherwise be silently rejected by the agent. - _createContext throws + frees the slot if a NativeSpanContext is passed as fields.context — re-wrapping would either leak the slot or duplicate the span. exporters/native/index.js — NativeExporter. - Batches raw span objects (not pre-formatted msgpack) for chunked export via NativeSpansInterface.flushSpans. - Atomic setUrl(): parse first, only assign this._url after the native setAgentUrl succeeds. - In-flight serialization: a second flush() while the first is unresolved buffers spans rather than starting a parallel send. Drains _pendingSpans on both success and rejection. - beforeExit handler registered on the dd-trace shared handler set (with process.once fallback), preventing listener leaks on repeated tracer construction. The subsystem compiles in isolation but is not wired into any tracer or exporter path yet — see the follow-up integration commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…, and OTel bridge
Wires the native subsystem into the existing tracer paths. The tracer
constructs the native pipeline whenever libdatadog is available; on
platforms where the native module can't be loaded, the tracer logs a
warn and falls back to the JS implementation.
opentracing/tracer.js:
- Lazy getNativeModule() so installs whose tracer never starts don't
pull libdatadog into memory.
- Native init in the DatadogTracer constructor constructs
NativeSpansInterface + NativeExporter + SpanProcessor with the
native interface attached when libdatadog is available.
- Warns once at init when libdatadog is unavailable so the JS
fallback isn't silent.
- Warns once at init when DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED
is set together with native mode (the JS-side spanFormat() path is
what emits process tags; the native exporter doesn't yet).
span_processor.js:
- Detects native mode via the nativeSpans constructor arg.
- _sampleNative: runs JS-side sampling (manual overrides first,
otherwise the standard priority sampler), then mirrors the
resulting priority/mechanism into native storage via
_syncSamplingToNative.
- getNativeOpCode is a lazy resolver; caches only on non-null,
log.errors once and returns null if OpCode is missing so the
caller short-circuits instead of NPE-ing in the sampling hot path.
opentelemetry/span.js:
- The OTel-API bridge constructs a NativeDatadogSpan instead of a
DatadogSpan when _tracer._nativeSpans is set; both pass to
super(ddSpan) so BridgeSpanBase still owns the OTel surface.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds 86 tests across five native specs plus targeted updates to span_processor.spec.js, covering the native subsystem and its integration points. - integration.spec.js: end-to-end against the real NativeSpansInterface (skipped when libdatadog is unavailable on the platform) — tracer wiring, full span lifecycle, double-finish protection, parent->child via active scope, service/resource/type via tracer.trace, error propagation, inject+extract round-trip. - native_spans.spec.js: queue methods, change-buffer detach-safety, flushChangeQueue rethrow + state reset, flushSpans rejection-path recovery, getStringId WASM-first ordering rollback, atomic setAgentUrl swap (including preservation on ctor failure). - span.spec.js: NativeDatadogSpan native-only behavior — combined queueCreateSpan op on construction, no-double SetName on init, slot-free + throw on duplicate-NativeSpanContext wrapping, syncOneTagToNative / syncToNativeOnly call assertions, priority short-circuit on tag application, SetDuration on finish. Behavior inherited from DatadogSpan is exercised by packages/dd-trace/test/opentracing/span.spec.js. - exporter.spec.js: in-flight serialization, drain on success and rejection, beforeExitHandlers registration. - span_context.spec.js: setTag side-effects to the WASM pipeline (covering SetServiceName / SetResourceName / SetType / SetError / SetMetaAttr / SetMetricAttr / SetTraceMetaAttr / SetTraceMetricsAttr / SetTraceOrigin), _syncNameToNative, nativeSpanId getter. The strengthened "should reset queue state when prepareChunk throws" test isolates the catch arm by stubbing flushChangeQueue to a no-op so the only observable cleanup path is the flushSpans catch — without this, the success-path reset inside flushChangeQueue would mask whether the catch arm runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Non-runtime support for the native-spans feature. Benchmarks: - benchmark/sirun/native-spans/* — creation, tagging, parent-child, pipeline, get-tag, and verify scenarios for the native pipeline. - benchmark/sirun/spans/* — small adjustments to align with the new per-span hot paths. Build/test plumbing: - package.json: test:trace:core glob includes the new `native` directory. - test/setup/core.js: clears OTEL_EXPORTER_OTLP_* env vars at suite start so plugin tests can stub http.request without observability tooling shells (Claude Code, etc.) hijacking traces through OTLP. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`@DataDog/libdatadog` moves out of `optionalDependencies` into `dependencies` — native spans are the only supported span pipeline going forward, and the WASM runtime that backs them must always be installed. With the optional install path gone, the JS-side fallback exporter, the JS-side span/stats formatter, and the OTLP-trace fallback exporter are all unreachable; this change deletes them and collapses the remaining call sites onto the native path. The dependency change is a stopgap. `@DataDog/libdatadog@0.9.3` is ~36 MB unpacked due to the bundled native binary set, which is what motivated optional-dep status originally. A smaller alternative is being explored elsewhere. Production modules deleted (no callers outside the JS-fallback branch): - packages/dd-trace/src/exporter.js (the JS-exporter resolver) - packages/dd-trace/src/exporters/agentless/ (index.js + writer.js) - packages/dd-trace/src/exporters/log/ - packages/dd-trace/src/exporters/span-stats/ (index.js + writer.js) - packages/dd-trace/src/span_format.js (JS-side span formatter) - packages/dd-trace/src/span_stats.js (JS-side stats processor) - packages/dd-trace/src/opentelemetry/trace/ (OTLP-trace exporter: index.js + otlp_http_trace_exporter.js + otlp_transformer.js) Production fallback branches removed: - packages/dd-trace/src/opentracing/tracer.js — drop the `if (getNativeModule().available)` outer gate, the surrounding try/catch, the JS-side `Exporter` + `SpanProcessor` construction, the `OTEL_TRACES_EXPORTER === 'otlp'` branch, and the unused `getExporter` import. - packages/dd-trace/src/opentelemetry/span.js — drop the JS `DatadogSpan` branch in the constructor; always build a `NativeDatadogSpan`. - packages/dd-trace/src/span_processor.js — drop the `_nativeSpans === null` branch in `sample()`, the `useJsFormatter` branch and JS-stats fallback in `process()`, the `SpanStatsProcessor` setup in the constructor, the `_stats` field, and the `spanFormat` import. The `nativeSpans` parameter is now required. - packages/dd-trace/src/native/index.js — `@DataDog/libdatadog` becomes a top-level require. Pipeline loading is deferred to first access (via a lazy `getPipeline()` helper) so importing this module from a unit test doesn't require a working pipeline binary, but any use throws hard if the pipeline can't load. The `available` boolean is gone. Test files deleted (one-to-one with deleted production): - packages/dd-trace/test/exporter.spec.js - packages/dd-trace/test/exporters/agentless/ (entire dir) - packages/dd-trace/test/exporters/log/ - packages/dd-trace/test/exporters/span-stats/ - packages/dd-trace/test/span_format.spec.js - packages/dd-trace/test/span_stats.spec.js - packages/dd-trace/test/opentelemetry/traces.spec.js Test files updated: - packages/dd-trace/test/span_processor.spec.js — rewritten around the native-only path; now seeds `nativeSpans` and `trace.tags` so `_sampleNative` / `_addDecisionMaker` have valid inputs, and asserts raw-span export instead of spanFormat output. Switches to `proxyquire.noCallThru()` so stubbing `./native` doesn't trigger the real pipeline load. - packages/dd-trace/test/opentracing/tracer.spec.js — rewritten to stub `NativeSpansInterface`, `NativeDatadogSpan`, and `NativeExporter` via proxyquire instead of asserting on the old AgentExporter / JS-exporter selection path. - packages/dd-trace/test/opentelemetry/span.spec.js — drops the now-deleted `span_format` import; the link- and exception-format assertions read the span context tags directly (the native span path serializes `_dd.span_links` / `ERROR_*` onto the context during finish). - packages/dd-trace/test/native/integration.spec.js — removes the `(skipped)` branch and de-indents the unconditional describe block. - packages/dd-trace/test/native/native_spans.spec.js and packages/dd-trace/test/native/span_context.spec.js — switch to `proxyquire.noCallThru()` so the in-test `./index` stubs no longer trigger proxyquire's callThru fallback into the real `src/native/index.js`. Bug also fixed in this change: `NativeDatadogSpan._addTags` was using `for-in`, which silently skipped Symbol-keyed entries like `IGNORE_OTEL_ERROR` (set by the OTel bridge's `applyOtelStatus`). The JS `DatadogSpan` parent uses `Object.assign`, which handles Symbols; the native subclass now does the same. `syncToNativeOnly` in `span_context.js` switches from `for-in` to `Object.keys` for the same reason and to honor the project's no-`for-in` rule. Notes: - `experimental.exporter` stays in the config schema: `ci_plugin.js` still reads it for CI worker-framework detection, and `_DD_APM_TRACING_AGENTLESS_ENABLED` still writes it (the write is a no-op now, but doesn't hurt). - Native span context already mirrors error/span-link encoding behaviorally; comments referencing `span_format.js` in `src/native/span.js` and `src/native/span_context.js` are left in place as historical references. - `Span` import (`./span` = JS DatadogSpan) is kept in `src/opentracing/tracer.js`: `NativeDatadogSpan` extends it, and `inject()` uses an `instanceof Span` check that still has to work for native spans. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The legacy agent writer publishes `dd-trace:exporter:first-flush` on its first flush; the native exporter introduced in this branch never did, so subscribers (notably the aborted-integrations log) never fire under the native span pipeline. Publish exactly once on the first successful native flush, gated on a private flag. The rejection path stays silent so transient agent errors don't trip the signal. Signed-off-by: Bryan English <bryan.english@datadoghq.com>
The protocol-version block in #applyEnvironment flipped OTEL_TRACES_EXPORTER from 'otlp' to 'none' whenever DD_TRACE_AGENT_PROTOCOL_VERSION was set. It originally guarded the JS-side OTLP traces exporter, which has been removed alongside the move to the native-span pipeline. With no OTLP exporter to disable, the only remaining effect was suppressing the OTel-sampler default for users who opted into 'otlp' on a non-default agent protocol version. Drop the block. The sampler-default branch downstream now fires consistently whenever OTEL_TRACES_EXPORTER='otlp' is set, regardless of DD_TRACE_AGENT_PROTOCOL_VERSION.
After the JS span pipeline was removed, two pre-existing comments in master-side files still named the deleted `span_format.js` as if it were live code: - `test/opentelemetry/context_manager.spec.js`: rationale for the numeric-startTime assertion attributed to span_format's `Math.round(startTime * 1e6)`. Reworded to point at the downstream ms-conversion generally; the test still guards the same shape. - `src/service-naming/extra-services.js`: comment claimed the cache exists for span_format's per-span hot path. The whole module has no production caller now (only tests + the global mocha clear hook); noted that explicitly so a future reader knows the cache is currently dormant rather than serving a hidden caller.
span_format.js previously copied _dd.rule_psr, _dd.limit_psr, and _dd.agent_psr from context._trace[KEY] onto the root span at format time. The native pipeline dropped that path. The native WASM pipeline supports these via SetTraceMetricsAttr (OpCode 11), which writes into trace.metrics and copies onto the chunk root span at flush. This restores the wiring by queuing SetTraceMetricsAttr ops for each sampling-decision metric present on spanContext._trace.
Native pipeline expects the caller to emit _dd.span_sampling.{mechanism,rule_rate,max_per_second} per-span metrics. These were previously set by span_format.setSingleSpanIngestionTags(), which was removed when the native pipeline replaced span_format.
Emit the three metrics inline in SpanSampler.sample() via queueBatchMetrics on the native spans interface, in the same location where _spanSampling is stamped.
registerExtraService was previously called from span_format.extractTags() (per span at format time). With the JS format pipeline removed in favor of the native WASM exporter, the extra_services Set was always empty and remote-config could no longer see per-service registrations, impacting AppSec service routing and other RC-driven service routing. Restore by calling registerExtraService in SpanProcessor.process() while iterating finished spans before export, matching the original timing.
This removes the unused _DD_APM_TRACING_AGENTLESS_ENABLED environment variable and all associated dead code. The feature was experimental and never fully shipped. The primary effect is removing the config block from `packages/dd-trace/src/config/index.js`.
AGENTS.md disallows async/await in production code outside test files
and worker threads (`packages/dd-trace/src/debugger/devtools_client/`).
`flushSpans` was async and awaited `_state.sendPreparedChunk()` inside
a try/catch.
Rewrite as a non-async function that returns a Promise via .then() /
.catch(). Behavior is identical:
- Same return shape: Promise<string> resolving to the agent response,
or 'no spans to flush' for empty slots.
- Same cleanup on either prepareChunk-throw or sendPreparedChunk-rejection:
resetChangeQueue + #checkDetach + log.error('Error flushing spans to
agent:', e), then propagate the error.
Test files keep their `await flushSpans()` calls; `await` works on any
thenable, and tests are exempt from the no-async/await rule.
The native span pipeline removed the JS span formatter (span_format.js) which previously added _dd.tags.process to the local root span. Restore it by adding the tag in the native exporter's #syncTraceTags method, guarded by DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED. Tag value comes from process-tags/index.js.
Old span_format.js automatically marked spans with span.kind (except internal) as measured so the agent computes metrics. This was lost when the JS formatter was removed. Add 'span.kind' to SPECIAL_KEYS to bypass the fast path, and emit SetMetricAttr for _dd.measured in the switch case.
The native span_context setTag path diverged from the deleted JS formatter (span_format.js) in three ways: - Plain object tag values were stringified to "[object Object]" instead of being flattened one level into key.prop entries. - NaN number metrics were emitted as f64 NaN; the old formatter dropped them entirely. - tracer.js warned that DD_EXPERIMENTAL_PROPAGATE_PROCESS_TAGS_ENABLED was unsupported, but the native exporter already emits _dd.tags.process on the local-root span. Remove the stale, contradictory warning. Introduce a shared appendTag helper mirroring the legacy addTag coercion (strings to meta, finite numbers to metrics, booleans to 0/1, plain objects flattened one level, arrays/Buffer/URL stringified as meta) and route all four value-handling sites through it to avoid divergence.
The native exporter discarded the value resolved by flushSpans. That value is the agent's response body, which carries rate_by_service used for adaptive (agent-driven) priority sampling. Without it the JS sampler never adapted to agent feedback and fell back to static rates \u2014 a regression from the legacy AgentWriter (which called prioritySampler.update(rate_by_service) on every response). The native sendPreparedChunk already surfaces this body to JS: it resolves 'unchanged' when the rates payload-version header matches the prior flush (no body), otherwise the raw JSON body. Parse the latter and forward rate_by_service to the priority sampler, swallowing/logging any malformed response so it never disrupts the flush cycle.
…rotocol The native change-buffer protocol moved from slot-indexed ([opcode:u64][slotIndex:u32]) to span_id-addressed ([opcode:u16][span_id:u64]) in the @DataDog/libdatadog bindings built against libdatadog main. Port the JS native layer to match. src: - op header: u16 opcode + the 8-byte LE span id (_nativeSpanId) as the handle; drop the u32 slot index and all slot allocation (allocSlot/freeSlots/_slotIndex removed across span_context, span, span_processor, span_sampler, native exporter) - CreateSpan: span_id moves to the header; new segment_id arg (trace_id u128, segment_id u64, parent_id u64, name_id u32, start i64). One segment_id per local trace, allocated on the shared _trace object and reused by children (required: native chunk flush keys by segment) - flush chunk buffer carries u64 span ids (8B) instead of u32 slots - batch meta/metric ops re-headered the same way tests (native_spans, span, span_context, exporter, span_sampler specs): - updated to the span_id protocol: u16 opcode offsets, span_id (Uint8Array) handle, segment allocator, queueCreateSpan signature, span-id flush arrays, and _nativeSpanId mock handles in the sampler spec. Verified: lint clean; the four test/native specs pass (67 tests); and an end-to-end run against the real span_id wasm (create root+child, Set*/Batch ops, flush) had the Rust decoder accept the chunk and emit a 460-byte payload. span_sampler.spec.js is ported + lint-clean but exercised by CI (it needs the built vendor/dist, unavailable locally).
meta_struct (AppSec, Code Origin, Dynamic Instrumentation) was dropped on the native path \u2014 there was no binding to set it, so structured per-span data never reached the agent. Add a setMetaStruct wrapper on NativeSpansInterface that drains the change queue first (the WASM binding flushes internally, so the JS-side queue offsets must stay in sync) and forwards the value to the new WasmSpanState.setMetaStruct, folding the 8-byte handle big-endian to the numeric span id. NativeDatadogSpan.finish now msgpack-encodes each qualifying meta_struct entry and forwards it, matching the legacy encoder's map<string, bin> wire shape and value filter (string/number/non-null object). Requires the companion WasmSpanState.setMetaStruct binding in libdatadog-nodejs.
A `unix://` agent URL already flows through to libdatadog's exporter and works once the wasm transport honors the socket path. Windows named pipes are represented as `unix://./pipe/...` (the legacy exporter's form), but ddcommon's parse_uri expects the `windows:` scheme for pipes. Normalize the agent URL in the single WasmSpanState construction path so the pipe form is rewritten to `windows://./pipe/...`; UDS and http(s) URLs pass through unchanged.
The native exporter emitted no tracer health metrics, so operators lost the datadog.tracer.node.exporter.agent.* counters the legacy AgentWriter produces (libdatadog's telemetry/health-metrics are native-only, cfg(not wasm32), and weren't wired from JS). Emit, around each native flush, matching the legacy metric names: - .requests on send - .responses on success - .errors + .errors.by.name (+ .errors.by.code) on failure .responses.by.status is intentionally omitted \u2014 sendPreparedChunk does not surface the HTTP status (libdatadog owns the transport). Covered by two new exporter.spec.js tests.
When DD_TRACE_NATIVE_SPAN_EVENTS is enabled (matching the legacy 0.4 encoder's gate), serialize each span event through the new native addSpanEvent setter so it lands in libdatadog's top-level v0.4 span_events field with typed attributes \u2014 no truncation. When the flag is off, keep the existing (lossy) _dd.span_events meta-tag fallback. Attributes are encoded into the flat little-endian buffer the pipeline crate decodes (per-value type tags String=0, Boolean=1, Integer=2, Double=3, Array=4); integer-valued numbers go out as i64, the rest as f64. The native_spans wrapper drains the change queue and refreshes detached views, mirroring setMetaStruct.
When DD_TRACE_AGENT_PROTOCOL_VERSION resolves to 0.5, the native exporter now fetches the agent /info (reusing agent/info.js) and, if the agent advertises /v0.5/traces, switches the native trace exporter to v0.5 via setUseV05(). v0.5 is opt-in and capability-gated: the v0.5 wire schema has no slot for meta_struct (or top-level span_events), so libdatadog silently drops them in v0.5 mode, matching dd-trace-js master's v0.5 encoder. Gating on both explicit config and agent advertisement avoids dropping that data for anyone who did not ask for v0.5. Negotiation is async; until it resolves the exporter stays on v0.4 (the safe default), and the selection is preserved across setAgentUrl().
…e channel, dm gating - span_context: skip SetError for error.type when IGNORE_OTEL_ERROR is set, so otel recordException() no longer flips the span error bit (only setStatus(ERROR) does). Mirrors span_format.js. - span: publish dd-trace:span:tags:update after native addTags so subscribers (e.g. the wall profiler's web-tag refresh) still fire on the fast path. - span_processor: write _dd.p.dm only for kept traces (priority >= AUTO_KEEP), matching the legacy priority sampler, instead of whenever a mechanism is set. Adds regression tests for all three.
… child trace id, dm gating - native_spans: decode the span handle little-endian in setMetaStruct/addSpanEvent to match how the change buffer keys spans (queueOp/queueCreateSpan copy the LE handle bytes). The prior big-endian fold attached meta_struct/span_events to the wrong native span for non-palindromic ids. - span: child and continued spans build the full 128-bit native trace id from the shared _dd.p.tid + the 64-bit id (buildNativeTraceId), instead of inheriting the 64-bit id and letting queueCreateSpan zero-pad the high bits — which recorded children under a different trace id than the root. - span_processor: _addDecisionMaker gates on priority >= AUTO_KEEP (was the wrong '>= 0'), so dropped/auto-rejected traces no longer get a _dd.p.dm tag. Regression tests added for all three.
When OTEL_TRACES_EXPORTER=otlp, configure the native exporter to send traces
over OTLP HTTP through libdatadog (which maps its internal traces to OTLP)
instead of to the Datadog agent, from the resolved
OTEL_EXPORTER_OTLP_TRACES_{ENDPOINT,PROTOCOL,HEADERS} config. This restores
OTLP export on the native path without a JS-side OTLP exporter.
- NativeSpansInterface gains setOtlpEndpoint/setOtlpProtocol/setOtlpHeaders,
forwarding to the wasm binding and persisting across setAgentUrl rebuilds
(mirroring setUseV05).
- NativeExporter configures OTLP synchronously at construction; OTLP takes
precedence over v0.5 (the agent path is bypassed), and an unsupported
protocol (e.g. grpc) is caught and falls back to the native default.
Address review-until-green feedback on the OTLP exporter wiring: - Consistent setter ordering: setOtlpEndpoint/setOtlpHeaders now forward to the native state before persisting (matching setOtlpProtocol), so a value the native layer rejects is never persisted and re-applied on a setAgentUrl rebuild. - Guard #configureOtlp against a missing OTLP endpoint (warn and skip rather than forwarding undefined). - Tests: protocol-rejection is not persisted/re-applied across setAgentUrl; empty headers map is a no-op; protocol-default leaves protocol/headers unset; grpc fallback and missing-endpoint both warn.
libdatadog's wasm binding now reports a fatal exporter-build failure (bad config \u2014 building is one-shot and unrecoverable) as a NativeExporterBuildError. Previously such a failure rejected every flush with 'exporter builder already consumed' and the exporter kept retrying, spamming errors and never recovering. Detect that error, disable the exporter (drop buffered spans, stop the flush timer, log once), and make export()/flush() no-ops afterwards.
0.11.0 is the first release that ships the `pipeline` wasm prebuild (the native-spans binding), so the native exporter can load it. Unblocks the CI that was failing with 'Could not find a pipeline binary' on 0.9.3.
Reconcile the native-spans branch with 359 commits of master drift.
Preserve the branch's native-only architecture:
- keep the removal of the legacy JS trace-export path (span_format,
span_stats, exporter.js, exporters/{agentless,span-stats},
encode/span-stats) and the OTLP SDK exporter (opentelemetry/trace);
the native pipeline replaces them
- restore exporters/agent (master's version) since profiling still
depends on it
- keep the native span_processor (raw spans to the native exporter, no
JS formatting/stats) over master's JS-formatting path
- keep native span creation in startSpan while adopting master's
post-creation ctx.setTag service tagging
- drop master's OTLP-vs-protocol-version config guard; native OTLP
coexists with the agent protocol version
- keep libdatadog a hard dependency (the native pipeline is required)
Adapt to master renames: the msgpack MsgpackEncoder class becomes the
functional encode(), stats.enabled becomes
stats.DD_TRACE_STATS_COMPUTATION_ENABLED, and drop tests for the removed
JS span-formatting path.
`TracerProvider.forceFlush()` called `exporter.flush()` unconditionally. In a Lambda with neither the extension layer nor the mini agent the tracer selects the stdout exporter, which writes synchronously and implements only `export()`, so any OpenTelemetry user calling `forceFlush()` there got a synchronous TypeError and the active span processor was never flushed either. Reported by Codex review as P2.
The v0.4 encoder runs `normalizeSpan` on every span as it encodes (`encode/0.4.js` selects it as the per-span formatter), so the JS pipeline never ships a span missing the intake defaults or exceeding the 100-character caps on service, name and type. The native path wrote `formatted.name` / `.service` / `.type` straight into WASM, making it the only pipeline that could emit un-normalized core fields — so a high-cardinality route name went out at full length. Apply the same pass at the native write, after the stats snapshot, which matches the legacy ordering where normalization happens at encode time rather than at finish. Reported by Codex review as P2. Note the report also mentions the 5,000 character resource cap; that is `truncateSpan`, which the v0.4 agent path does not apply either (only the electron and agentless encoders do), so it is deliberately left alone.
The `meta.events` JSON fallback exists for agents that cannot read the native `span_events` field, gated on `DD_TRACE_NATIVE_SPAN_EVENTS`. That gate is about the agent protocol, so with `OTEL_TRACES_EXPORTER=otlp` and the default flag value every event reached the collector as a JSON string attribute instead of a structured OTLP event, breaking consumers of exception data. The deleted OTLP transformer converted events regardless of the flag. Take the native path whenever the destination is OTLP. Reported by Codex review as P2.
Keep the existing flush timer authoritative after an in-flight native send and trigger an early flush at 2,000 pending spans. The native exporter waited for the two-second interval before its first send, then bypassed batching after every settlement. On the Express/PostgreSQL workload, bounded batching reduced CPU/request by 18.6%, increased throughput by 25.1%, and reduced RSS from 1,344 MiB to 366 MiB. - Run the native exporter unit tests. - Verify every changed production branch with c8. - Run the full repository lint. - Run three fresh-process 50,000-request Express/PostgreSQL trials with exact trace and query counts.
All spans in a trace share an immutable ID, but the native path rebuilt its 16-byte representation for every child. Reusing the shared representation reduced buildNativeTraceId self-time from 94.6 to 30.8 ms in a 50,000-request Express/PostgreSQL profile and its isolated seven-span path from 1,081 to 169 ns/trace.
Assigning and deleting the construction-time name hook put every native span context on a slow object shape even though final synchronization already owns name writes. Removing the stale hook reduced CPU per request from 112.41 to 99.77 µs and raised throughput from 11,039 to 12,175 requests/s in the Express/PostgreSQL workload; isolated construction fell from 75.1 to 6.4 ns/span.
## Summary Keep resource names intact in native fast final sync, matching the JS v0.4 and v0.5 encoders. ## Why The tracer applied the agent's 5,000-character normalization limit before native export. Long SQL resources then differed between exporters and failed the existing PostgreSQL wire assertions. ## Test plan - ./node_modules/.bin/mocha packages/dd-trace/test/native/span_context.spec.js - Existing PostgreSQL long-query cases for pg 8.0.3 and 8.22 (2 passing)
## Summary Mirror the canonical formatter's base-service inference in native fast final sync and remove the now-redundant configured-service field from native contexts. ## Why The fast path registered an overridden service but skipped _dd.base_service. Raw spans and WASM output therefore lost the configured service whenever a span selected a different one. ## Test plan - Native tracer and span-context suites (45 passing) - Native plugin wire assertions for base-service propagation (3 passing)
## Summary Change-queue flushing consumed 95.8% of the deferred-finish profile and kept the eight-sample CI variant running when the 30-minute job expired. ## Why The benchmark is meant to isolate span construction and finish, but it applied and exported every queued native mutation. Discarding those mutations reduced the same 250,000-span process from 23.11 s to 0.60 s. Native event samples still drain because libdatadog applies events directly. ## Test plan - Run all span variants through three fresh sirun matrices. - Run changed-line coverage and full lint.
Keeping span mutation in JS preserves AppSec live-span reads and finish-time mutations while removing the per-tag WASM mutation stream. Finalized v0.4 payloads are encoded once and handed to libdatadog. Stats, v0.5, and OTLP still decode where their contracts require span access. On Node 24.18.0 / V8 13.6 with 100,000 spans, seven alternating trials with the best and worst removed, total CPU changed from 432.31889179999996 to 188.20740819999997 ms for plain spans, 661.9301330000001 to 459.0956 ms for tag-heavy spans, and 3053.0080915999997 to 509.23610859999997 ms for AppSec-shaped spans. Drive-by fix: * Preserve lone UTF-16 surrogates in legacy span-event JSON.
|
…-native-spans-master
Agentless tracing previously stayed on the JS exporter, so libdatadog could not normalize or obfuscate the final payload. Native selection now requires an explicit binding capability, preserves the JS fallback for older packages, and fails closed when agentless configuration is invalid.
Overall package sizeSelf size: 8.24 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 445.14 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #9838 +/- ##
==========================================
- Coverage 98.53% 93.49% -5.04%
==========================================
Files 978 936 -42
Lines 144181 138543 -5638
Branches 12443 10566 -1877
==========================================
- Hits 142069 129535 -12534
- Misses 2112 9008 +6896 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Keeping mutation in JS preserves AppSec live-span reads and finish-time mutations without the per-tag WASM mutation stream. Finalized v0.4 trace batches are encoded once before libdatadog transport. Stats, v0.5, and OTLP still decode where their contracts require span objects.
On Node 24.18.0 / V8 13.6, 100,000-span trials reduce CPU from 432.319 to 188.207 ms for plain spans, 661.930 to 459.096 ms for tag-heavy spans, and 3,053.008 to 509.236 ms for AppSec-shaped spans.
A traced Express Hello World run against current master still uses 2.44% more server CPU and has 2.13% lower throughput. The 2,000-span cap creates 75 agent posts per 50,000 requests instead of 11 or 12, so batching remains part of this draft.
This draft depends on the matching libdatadog and libdatadog-nodejs changes.