Skip to content

feat: decode binary/protobuf payloads with messageType (LOE)#3356

Draft
rossnelson wants to merge 7 commits intorefactor-payload-componentfrom
decode-binary-protobuf-payloads
Draft

feat: decode binary/protobuf payloads with messageType (LOE)#3356
rossnelson wants to merge 7 commits intorefactor-payload-componentfrom
decode-binary-protobuf-payloads

Conversation

@rossnelson
Copy link
Copy Markdown
Collaborator

Summary

LOE prototype for browser-side decode of System Nexus payloads, where the gRPC envelope (SignalWithStartWorkflowExecutionRequest, future WaitForExternalWorkflow…, StandaloneActivity…, etc.) arrives in workflow history as a single Payload with metadata.encoding = binary/protobuf and metadata.messageType naming a fully-qualified temporal.api.* type.

Adds a Phase 1.5 dispatch in parseRawPayloadToJSON:

encoding === 'binary/protobuf' && metadata.messageType
  → @temporalio/proto: lookupType(messageType).decode(bytes).toObject()
  → recursively decode any nested {metadata, data} Payloads in the result
  → return as if it had been encoding 'json/plain' all along

Every existing renderer (event card, JSON view, compact row, copy/export) inherits the typed view for free. Adding the next system Nexus operation costs zero code — @temporalio/proto already ships every workflowservice message.

Demo

Repro harness lives in temporalio/nexus-endpoint (sibling repo): pinned temporal server + sdk-python spk/signal-with-start test that produces a real SignalWithStartWorkflowExecutionRequest in workflow history.

Before — what every renderer saw before this patch for nexusOperationScheduledEventAttributes.input:

{
  "metadata": {
    "encoding":    "binary/protobuf",
    "messageType": "temporal.api.workflowservice.v1.SignalWithStartWorkflowExecutionRequest"
  },
  "data": "CgdkZWZhdWx0EhhzeXN0ZW0tbmV4… (1068 base64 chars)"
}

After — same render path, no other changes:

{
  "namespace":     "default",
  "workflowId":    "system-nexus-workflow-id",
  "workflowType":  { "name": "EchoWorkflow" },
  "taskQueue":     { "name": "969f0ffd-…" },
  "input":         { "payloads": [ { "driver_claim": { }, "driver_name": "test-driver" } ] },
  "signalName":    "test-signal",
  "signalInput":   { "payloads": [ { "driver_claim": { }, "driver_name": "test-driver" } ] },
  "memo":          { "fields": { "memo-key": { "driver_claim": { }, "driver_name": "test-driver" } } },
  "userMetadata":  { "summary": { }, "details": { } },
  
}

Note the inner Payloads (the user's signal args, memo, etc.) are also decoded — the recursive pass in this PR feeds nested {metadata, data} nodes back through parseRawPayloadToJSON, so the existing JSON-decode path handles them transparently.

Visible UI surfaces (verified live)

  • Event detail panel — Input / Result typed JSON
  • Compact event row in the events table — same shape inline
  • "Copy as JSON" / export — same shape

Open questions for review

  1. CSP — second commit relaxes script-src to add 'unsafe-eval', because protobufjs builds per-message decoders via Function(). Two paths:
    • Keep this commit (accept unsafe-eval in production)
    • Drop this commit and migrate to @bufbuild/protobuf (no eval, smaller, ESM-native; ~2 days)
  2. Bundle size@temporalio/proto is already a dep but only used as types today. Worth a pnpm build before/after to see the runtime delta. If meaningful, switch to dynamic await import('@temporalio/proto') inside the decode helper.
  3. ./atob landmine — the local atob does UTF-8 decoding via decodeURIComponent, which corrupts raw protobuf bytes. The patch uses globalThis.atob directly. Worth either documenting that on ./atob.ts or extracting a rawBase64ToUint8Array helper for future binary work.
  4. UX — should we render Nexus operations as their inner type (e.g. "Signal With Start" as a first-class event row) rather than "Nexus Operation Scheduled" with a typed Input panel? Notion doc is silent. Worth a designer pass.

What this isn't

  • A production-ready feature. It's an LOE prototype. Drafting to share the working approach.
  • Anything beyond binary/protobuf payloads. Other encodings unchanged.
  • A Nexus presentation refactor. The event card still says "Nexus Operation Scheduled".

Test plan

  • pnpm test src/lib/utilities/decode-payload.test.ts -- --run — 27/27 passing (added 2)
  • pnpm check — 0 errors, 84 pre-existing warnings
  • pnpm lint — 0 errors, 210 pre-existing warnings
  • End-to-end: real workflow with SignalWithStartWorkflowExecutionRequest + nested external-storage payloads decoded in browser, screenshots in conversation
  • pnpm build size comparison — TODO before promoting beyond LOE

Notion

System Nexus Endpoint — the page's outstanding "UI" question is whether the browser can decode without a proto library. Answer: yes, it already can; this PR shows how.

Add a Phase 1.5 dispatch in parseRawPayloadToJSON: when a Payload's
metadata says encoding=binary/protobuf and carries a messageType, look
the type up by full name in @temporalio/proto's static namespace and
return the typed JSON. After the outer decode, walk the result tree
and recursively decode any nested {metadata, data} Payload nodes
through the same pipeline so user-side payloads (json/plain, json/
protobuf, json/external-storage-reference, etc.) come out fully
parsed. Unknown messageTypes and decode errors fall through to the
existing path so behavior for non-binary/protobuf payloads is
unchanged.

This is the proposal for handling System Nexus operations like
SignalWithStartWorkflowExecutionRequest, where the gRPC envelope
arrives as binary protobuf and the inner Payloads carry user data.

Tests cover round-tripping a real SignalWithStart fixture and the
graceful fallback for an unknown messageType. 27/27 passing.

Caveat: protobufjs builds its decoders with Function() and so needs
'unsafe-eval' in the script-src CSP. That's relaxed in a separate
commit so reviewers can drop it before merge if we want a different
proto runtime (@bufbuild/protobuf has no eval).
@temporalio/proto uses protobufjs, which compiles per-message
encoders/decoders via the Function() constructor. With the existing
'strict-dynamic' CSP and no 'unsafe-eval', binary/protobuf payload
decode silently fails in the browser.

Relax the dev/auto CSP to permit unsafe-eval so the new decode path
works end-to-end during the LOE evaluation. If the team decides to
ship browser-side proto decoding, the long-term option is to swap
@temporalio/proto for @bufbuild/protobuf (which doesn't use eval) and
drop this commit.

Open question for review: do we keep this, or migrate the runtime?
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 29, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
holocene Ready Ready Preview, Comment May 4, 2026 5:48pm

Request Review

rossnelson added 3 commits May 4, 2026 13:05
…-binary-protobuf

Moves lookupTemporalProtoType, base64ToUint8Array, looksLikeRawPayload,
recursivelyDecodeNestedPayloads, and the full decode path into a new
decode-binary-protobuf utility. The recursive walker now accepts a recurse
callback rather than calling parseRawPayloadToJSON directly, eliminating
the circular dependency. Adds unit tests covering successful decode, silent
null on unknown type, warn-on-throw, looksLikeRawPayload edge cases, and
the callback injection path.
…quests

Converts payload-decoder.svelte from {#await} to AbortController + $effect
so stale in-flight decodes cannot overwrite a newer result. Rewrites
codeServerRequest in data-encoder.ts to accept an optional AbortSignal and
retry transient errors (network/5xx) up to 3 attempts with 500ms/1000ms
delays, without retrying 4xx. Deletes the now-unused decode-payload-value
module (zero importers confirmed).
Adds a nexus-operation-registry that recognises the 4 system-level Nexus
operation types (StartWorkflow, SignalWorkflow, SignalWithStartWorkflow,
QueryWorkflow) and returns a human-readable descriptor with the embedded
input payloads. NexusOperationRenderer uses the registry to show the
embedded operation directly — users see "Signal With Start Workflow:
EchoWorkflow" and the decoded input, not the raw protobuf wrapper.
input-and-results-payload routes Nexus payloads through the renderer
automatically; all other content paths are unchanged. Adds 6 registry
tests covering the happy path and null returns for unknown types.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant