feat: decode binary/protobuf payloads with messageType (LOE)#3356
Draft
rossnelson wants to merge 7 commits intorefactor-payload-componentfrom
Draft
feat: decode binary/protobuf payloads with messageType (LOE)#3356rossnelson wants to merge 7 commits intorefactor-payload-componentfrom
rossnelson wants to merge 7 commits intorefactor-payload-componentfrom
Conversation
Add a Phase 1.5 dispatch in parseRawPayloadToJSON: when a Payload's
metadata says encoding=binary/protobuf and carries a messageType, look
the type up by full name in @temporalio/proto's static namespace and
return the typed JSON. After the outer decode, walk the result tree
and recursively decode any nested {metadata, data} Payload nodes
through the same pipeline so user-side payloads (json/plain, json/
protobuf, json/external-storage-reference, etc.) come out fully
parsed. Unknown messageTypes and decode errors fall through to the
existing path so behavior for non-binary/protobuf payloads is
unchanged.
This is the proposal for handling System Nexus operations like
SignalWithStartWorkflowExecutionRequest, where the gRPC envelope
arrives as binary protobuf and the inner Payloads carry user data.
Tests cover round-tripping a real SignalWithStart fixture and the
graceful fallback for an unknown messageType. 27/27 passing.
Caveat: protobufjs builds its decoders with Function() and so needs
'unsafe-eval' in the script-src CSP. That's relaxed in a separate
commit so reviewers can drop it before merge if we want a different
proto runtime (@bufbuild/protobuf has no eval).
@temporalio/proto uses protobufjs, which compiles per-message encoders/decoders via the Function() constructor. With the existing 'strict-dynamic' CSP and no 'unsafe-eval', binary/protobuf payload decode silently fails in the browser. Relax the dev/auto CSP to permit unsafe-eval so the new decode path works end-to-end during the LOE evaluation. If the team decides to ship browser-side proto decoding, the long-term option is to swap @temporalio/proto for @bufbuild/protobuf (which doesn't use eval) and drop this commit. Open question for review: do we keep this, or migrate the runtime?
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…-binary-protobuf Moves lookupTemporalProtoType, base64ToUint8Array, looksLikeRawPayload, recursivelyDecodeNestedPayloads, and the full decode path into a new decode-binary-protobuf utility. The recursive walker now accepts a recurse callback rather than calling parseRawPayloadToJSON directly, eliminating the circular dependency. Adds unit tests covering successful decode, silent null on unknown type, warn-on-throw, looksLikeRawPayload edge cases, and the callback injection path.
…quests
Converts payload-decoder.svelte from {#await} to AbortController + $effect
so stale in-flight decodes cannot overwrite a newer result. Rewrites
codeServerRequest in data-encoder.ts to accept an optional AbortSignal and
retry transient errors (network/5xx) up to 3 attempts with 500ms/1000ms
delays, without retrying 4xx. Deletes the now-unused decode-payload-value
module (zero importers confirmed).
Adds a nexus-operation-registry that recognises the 4 system-level Nexus operation types (StartWorkflow, SignalWorkflow, SignalWithStartWorkflow, QueryWorkflow) and returns a human-readable descriptor with the embedded input payloads. NexusOperationRenderer uses the registry to show the embedded operation directly — users see "Signal With Start Workflow: EchoWorkflow" and the decoded input, not the raw protobuf wrapper. input-and-results-payload routes Nexus payloads through the renderer automatically; all other content paths are unchanged. Adds 6 registry tests covering the happy path and null returns for unknown types.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LOE prototype for browser-side decode of System Nexus payloads, where the gRPC envelope (
SignalWithStartWorkflowExecutionRequest, futureWaitForExternalWorkflow…,StandaloneActivity…, etc.) arrives in workflow history as a singlePayloadwithmetadata.encoding = binary/protobufandmetadata.messageTypenaming a fully-qualifiedtemporal.api.*type.Adds a Phase 1.5 dispatch in
parseRawPayloadToJSON:Every existing renderer (event card, JSON view, compact row, copy/export) inherits the typed view for free. Adding the next system Nexus operation costs zero code —
@temporalio/protoalready ships every workflowservice message.Demo
Repro harness lives in
temporalio/nexus-endpoint(sibling repo): pinnedtemporalserver +sdk-pythonspk/signal-with-starttest that produces a realSignalWithStartWorkflowExecutionRequestin workflow history.Before — what every renderer saw before this patch for
nexusOperationScheduledEventAttributes.input:{ "metadata": { "encoding": "binary/protobuf", "messageType": "temporal.api.workflowservice.v1.SignalWithStartWorkflowExecutionRequest" }, "data": "CgdkZWZhdWx0EhhzeXN0ZW0tbmV4… (1068 base64 chars)" }After — same render path, no other changes:
{ "namespace": "default", "workflowId": "system-nexus-workflow-id", "workflowType": { "name": "EchoWorkflow" }, "taskQueue": { "name": "969f0ffd-…" }, "input": { "payloads": [ { "driver_claim": { … }, "driver_name": "test-driver" } ] }, "signalName": "test-signal", "signalInput": { "payloads": [ { "driver_claim": { … }, "driver_name": "test-driver" } ] }, "memo": { "fields": { "memo-key": { "driver_claim": { … }, "driver_name": "test-driver" } } }, "userMetadata": { "summary": { … }, "details": { … } }, … }Note the inner Payloads (the user's signal args, memo, etc.) are also decoded — the recursive pass in this PR feeds nested
{metadata, data}nodes back throughparseRawPayloadToJSON, so the existing JSON-decode path handles them transparently.Visible UI surfaces (verified live)
Input/Resulttyped JSONOpen questions for review
script-srcto add'unsafe-eval', because protobufjs builds per-message decoders viaFunction(). Two paths:unsafe-evalin production)@bufbuild/protobuf(no eval, smaller, ESM-native; ~2 days)@temporalio/protois already a dep but only used as types today. Worth apnpm buildbefore/after to see the runtime delta. If meaningful, switch to dynamicawait import('@temporalio/proto')inside the decode helper../atoblandmine — the localatobdoes UTF-8 decoding viadecodeURIComponent, which corrupts raw protobuf bytes. The patch usesglobalThis.atobdirectly. Worth either documenting that on./atob.tsor extracting arawBase64ToUint8Arrayhelper for future binary work.What this isn't
binary/protobufpayloads. Other encodings unchanged.Test plan
pnpm test src/lib/utilities/decode-payload.test.ts -- --run— 27/27 passing (added 2)pnpm check— 0 errors, 84 pre-existing warningspnpm lint— 0 errors, 210 pre-existing warningsSignalWithStartWorkflowExecutionRequest+ nested external-storage payloads decoded in browser, screenshots in conversationpnpm buildsize comparison — TODO before promoting beyond LOENotion
System Nexus Endpoint — the page's outstanding "UI" question is whether the browser can decode without a proto library. Answer: yes, it already can; this PR shows how.