feat(otel): connect durable orchestration spans across worker processes - #9794
feat(otel): connect durable orchestration spans across worker processes#9794chemystery09 wants to merge 4 commits into
Conversation
Persist one orchestration span identity per instance in a shared store so activities on any worker parent correctly, seed metadata from the HTTP span at startNew, and export a single orchestration span when the instance completes instead of one span per replay turn. Co-authored-by: Cursor <cursoragent@cursor.com>
Overall package sizeSelf size: 8.07 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.3.3 | 125.43 kB | 441.68 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad72846cff
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| try { | ||
| // Provided by the Azure Functions host, so it is never a tracer dependency. | ||
| // eslint-disable-next-line n/no-missing-require | ||
| const { TableClient } = require('@azure/data-tables') |
There was a problem hiding this comment.
When the app has AzureWebJobsStorage but has not installed @azure/data-tables, this require throws and getTableClient() permanently returns null. Since dd-trace does not declare that package (I only find this dynamic require in the repo), the Azure Table leg of the shared store is disabled for ordinary Durable Functions apps, leaving only per-worker tmp/cache and breaking the cross-worker parent resolution this change is meant to provide.
Useful? React with 👍 / 👎.
|
|
||
| function ensureOrchestrationMeta (instanceId, invocationContext, functionName) { | ||
| const traceContext = invocationContext?.traceContext | ||
| let meta = readOrchestrationSpanMetaSync(instanceId, traceContext) |
There was a problem hiding this comment.
When startNew runs on worker A and the orchestration first executes on worker B, the HTTP-seeded identity is only in Azure Table. This synchronous lookup checks only cache/file/tracestate, so worker B creates and publishes a fresh orchestration span ID/parent, often overwriting the seeded row; the exported orchestration span and subsequent activities no longer connect to the HTTP span.
Useful? React with 👍 / 👎.
| const { readOrchestrationSpanMetaSync } = require('./otel-orchestration-store') | ||
| const meta = readOrchestrationSpanMetaSync(instanceId, traceContext) |
There was a problem hiding this comment.
For durable activity handlers that are not AsyncFunction, wrapSyncWithTraceContext comes through this path, which only calls readOrchestrationSpanMetaSync. If the activity lands on a different worker and the orchestration metadata only exists in Azure Table, the span falls back to the Azure-internal traceparent instead of the orchestration span, so sync activities remain disconnected while async ones work.
Useful? React with 👍 / 👎.
| function writeMetaFileSync (instanceId, meta) { | ||
| const directory = getStoreDirectory() | ||
| fs.mkdirSync(directory, { recursive: true }) | ||
| fs.writeFileSync(getMetaFilePath(instanceId), JSON.stringify(meta)) |
There was a problem hiding this comment.
This write runs synchronously from startNew and orchestration setup/completion; if the temp store is unwritable/full or a caller-supplied instance ID produces an invalid path, the fs exception propagates out of the instrumentation and fails the user's function even though losing this cache should only degrade tracing. Catch/log and continue around the local store write.
Useful? React with 👍 / 👎.
| httpParentByInstance.set(key, normalized) | ||
| pendingHttpParentByTraceId.set(normalized.traceId, normalized) |
There was a problem hiding this comment.
Every successful startNew adds entries to these module-level maps, but there is no delete path when the instance is seeded, completed, or evicted. In long-lived Azure Functions workers that start many orchestrations, these maps retain one metadata object per instance/trace indefinitely, so tracer memory grows with total historical orchestrations rather than active work.
Useful? React with 👍 / 👎.
| META_CACHE.delete(instanceId) | ||
| deleteMetaFileSync(instanceId) |
There was a problem hiding this comment.
After completion this only clears the process cache and temp file, leaving the Azure Table row written by publishOrchestrationMetaSync() behind. In apps that purge and reuse custom/singleton instance IDs, another worker can later read the stale completed row from readOrchestrationSpanMetaAsync() and parent a new run's activities to the old span; high-volume apps also accumulate one extra table row per orchestration indefinitely.
Useful? React with 👍 / 👎.
| ensureTable() | ||
| .then(table => table.upsertEntity({ |
There was a problem hiding this comment.
The Azure Table upsert is detached from the HTTP startNew path, so the starter can return (and be frozen on a serverless plan) before the cross-worker row is written. If the orchestration is picked up by another worker in that window, or the process freezes before the promise runs, the worker cannot find the HTTP-seeded parent and creates a different orchestration identity; make the seed path await the shared-store persistence before startNew resolves.
Useful? React with 👍 / 👎.
| function getMetaFilePath (instanceId) { | ||
| return path.join(getStoreDirectory(), `${instanceId}.json`) |
There was a problem hiding this comment.
Durable custom instance IDs may be user-specified strings, and this uses them directly in path.join(). An ID containing path separators or .. can make the tracer write and later delete JSON outside dd-orchestration-spans (for example under another temp directory path), so encode or hash the ID before using it as a filename.
Useful? React with 👍 / 👎.
| partitionKey: TABLE_PARTITION_KEY, | ||
| rowKey: instanceId, |
There was a problem hiding this comment.
Durable instance IDs are only unique within a task hub, but the shared table key uses a constant partition and only instanceId as the row key. When two task hubs or function apps share AzureWebJobsStorage and use the same custom/singleton ID, one orchestration's metadata can overwrite the other's and activities can be parented to the wrong trace; include the task hub/app scope in the key.
Useful? React with 👍 / 👎.
| if (instanceId) { | ||
| completeOrchestrationSpan(TRACER_NAME, instanceId, invocationContext, functionName) |
There was a problem hiding this comment.
This completion path now runs even when invocationContext.df.isReplaying was true at entry (only ensureOrchestrationMeta() is gated). A replay of an already-started orchestration that reaches the end can therefore export and clear the synthetic orchestration span before a non-replay completion, causing duplicate or missing parent metadata; keep the previous replay short-circuit or gate completion on non-replay terminal execution.
Useful? React with 👍 / 👎.
…rents async Stamp orchestration start time on the first non-replay turn instead of at startNew, export one backfilled span for the full instance window, and wrap all durable activities with async parent resolution so sync handlers on other workers read the shared orchestration store. Co-authored-by: Cursor <cursoragent@cursor.com>
…vity times The prior first-turn stamp could run after replay resumed, which left activity spans starting before their orchestration parent in the waterfall. Seed startTime at HTTP startNew, record the earliest activity start in the shared store, and clamp export bounds to that window. Co-authored-by: Cursor <cursoragent@cursor.com>
Add unit tests for stampOrchestrationStartTime and resolveOrchestrationSpanBounds used by the backfilled orchestration span export path in #9794. Co-authored-by: Cursor <cursoragent@cursor.com>
…t bounds Always record the HTTP instance start at startNew (using the active span start time when available), track earliest activity starts even before shared meta exists, and merge both floors when exporting the backfilled orchestration span so activities no longer precede their parent in the waterfall. Co-authored-by: Cursor <cursoragent@cursor.com>
Add unit tests for stampOrchestrationStartTime and resolveOrchestrationSpanBounds used by the backfilled orchestration span export path in #9794. Co-authored-by: Cursor <cursoragent@cursor.com>
1 participant
Summary
Stacked on #9683. Fixes cross-worker trace parenting for Durable Functions on Azure.
Problem this solves: Azure DTF distributed tracing parents each invocation to extension-internal spans. Orchestrations also run later and on different workers than the HTTP trigger that called
startNew. Without shared state:What this PR does:
instanceIdin a shared store (in-process cache → local JSON → Azure TableDDAzureOrchestrationSpans)DurableClient.startNewso the orchestration span can parent under HTTP even when the orchestrator runs elsewherestartTimeanchored at HTTPstartNew,endTimeat completion, clamped to earliest activity start)All activity handlers use async parent resolution so sync activities on a different worker than the HTTP trigger can read the Azure Table store (not just local JSON).
Orchestration span timing (approach 1: backfill at completion)
We intentionally do not emit a live orchestration span on every replay turn. Instead, one span is backfilled when the instance completes, using metadata timestamps:
startTime— anchored atDurableClient.startNew(HTTP span start time when available), merged withrecordEarliestChildStartTimeat export viaresolveExportStartTimeendTime— instance completion timeThis fixes waterfall ordering where activity spans (live) appeared to start before their orchestration parent. The prior “first non-replay orchestrator turn” stamp was too late because replay turns skipped metadata updates until after the first activity had already exported.
Drawbacks of this approach (explicit trade-offs):
Date.now()bounds; extreme clock skew across workers could still produce small visual gaps (clamped on export).A future live orchestration span mode (approach 2) could address in-flight visibility but adds complexity across replay, worker scale-out, and agentless flush — out of scope here.
Required configuration (Function app)
Everything from #9683, plus:
Must be enabled
AzureWebJobsStorageDDAzureOrchestrationSpanstable used cross-worker@azure/data-tablesclient.startNew()PizzaParty→startNew('PizzaOrderOrchestration', …)startNewhook seeds orchestration span identity from the active HTTP spanOptional
DD_TRACE_AZURE_ORCHESTRATION_STORE_DIR$TMPDIR/dd-orchestration-spans)Still must be disabled (same as #9683)
DD_TRACE_AZURE_DURABLE_FUNCTIONS_ENABLEDfalseplugins: falsehost.json (unchanged from #9683)
Architecture (brief)
Changes
otel-orchestration-store.js,otel-orchestration-meta.js,otel-orchestration-export.js,otel-orchestration-http-link.js,otel-orchestration-registry.jsazure-trace-context.js— activity parent resolution from storeotel-azure-durable-functions.js/otel-azure-functions.js— one span per instance; async activity parent resolutionazure-durable-functions.js—DurableClient.startNewhookrecordHttpInstanceStartTime/recordEarliestChildStartTime/resolveExportStartTime— full-instance orchestration span windowTest plan
Verified — local Azurite storage (
durable-node, not DTS)Storage:
AzureWebJobsStorage=UseDevelopmentStorage=true(Azurite). Servicedurable-node-azurite-local, envlocal.POST http://localhost:7071/api/pizzaparty(PizzaOrderOrchestration: Prepare → Bake)PreparePizzaActivityandBakePizzaActivityparent underorchestration PizzaOrderOrchestrationhttp PizzaParty, not phantom rootstartNewtime and covers activity childrenExample trace (2026-08-13): instance
22d765967b474e87a1c4f54e50020e7d— https://ddserverless.datadoghq.com/apm/trace/b888f30374a9d9ce987b9b921f204eb4Pending — Azure Durable Task Scheduler (
durable-node-fx)Storage: DTS (
test-scheduler/ishara-node-hub), servicedurable-function-poc-node, envdev.POST https://durable-node-fx.azurewebsites.net/api/pizzaparty— orchestrations complete17862494001755642471FUNCTIONS_WORKER_PROCESS_COUNT=4locally or Azure scale-out)Known limitation: multi-worker activity parenting
This is not an Azurite vs DTS storage issue. The shared store path (local JSON → Azure Table via
AzureWebJobsStorage) is the same on local Azurite and Azure. What matters is worker topology:func start)b888f303…; DTS trace178624940017556424715825695498924434065:PreparePizzaActivityparented correctly (same worker as orchestrator);BakePizzaActivityorphaned (different worker)Locally,
FUNCTIONS_WORKER_PROCESS_COUNT>1can reproduce the same race; default single-worker dev runs do not exercise it.Root cause: orchestration meta is published to Azure Table asynchronously (errors swallowed); activity handlers async-read with a short retry budget (~300 ms). A second worker that starts before the row is visible gets no meta and parents to the Azure extension span instead.
Possible follow-ups (not in this PR)
dd=o:{orchestrationSpanId}at dispatch; helper exists)Recommended next step: hybrid — tracestate injection first, then await only the first table publish before the orchestrator's first yield.
Quick reference — full OTel-only stack (#9683 + #9794)