From 5ce175b557547714b9a84f6b0a512f7f312d8fcc Mon Sep 17 00:00:00 2001 From: bymyself Date: Sun, 23 Aug 2026 07:09:28 +0000 Subject: [PATCH 1/3] docs(adr): formalize workflow-to-execution-payload transform proposal Writes up Ben Cooley's 2026-08-21 proposal as ADR 0016. Reconstruction note is prominent; Ben should confirm before this is accepted. Fixes #15704 --- ...workflow-to-execution-payload-transform.md | 176 ++++++++++++++++++ 1 file changed, 176 insertions(+) create mode 100644 docs/adr/0016-workflow-to-execution-payload-transform.md diff --git a/docs/adr/0016-workflow-to-execution-payload-transform.md b/docs/adr/0016-workflow-to-execution-payload-transform.md new file mode 100644 index 00000000000..7a02a204d48 --- /dev/null +++ b/docs/adr/0016-workflow-to-execution-payload-transform.md @@ -0,0 +1,176 @@ +# 16. Workflow-to-execution-payload transform + +Date: 2026-08-23 + +## Status + +Proposed + +## Context + +> **Reconstruction note (Christian Byrne):** The proposal below is reconstructed +> from a Slack message in `#p-frontend-graph-improvements` posted +> 2026-08-22T08:33:50Z (thread `p1787387630437609`). Ben Cooley's verbal +> proposal from the 2026-08-21 API v2 discussion has no transcript — Fireflies +> has no record of August meetings, and a targeted search for attendees Ben, Alex, +> and Christian on 2026-08-21 around 2pm PT returned nothing. Sections clearly +> derived from that reconstruction are marked **[reconstruction]**. Ben should +> confirm or correct before this ADR is accepted. + +### The two formats + +ComfyUI operates with two JSON representations of a workflow: + +**Workflow format** (`ComfyWorkflowJSON`) — the save format. Contains the full +graph structure: nodes with positions and widget values, typed links, subgraph +definitions, metadata, and canvas state. This is what `.json` files and PNG +`workflow` metadata contain. Defined in +`src/platform/workflow/validation/schemas/workflowSchema.ts`. + +**Execution payload** (`ComfyApiWorkflow`) — the prompt sent to the server. +A flat dict keyed by node ID: `{ [nodeId]: { inputs, class_type, _meta } }`. +Link references are `[sourceNodeId, sourceSlot]` tuples. Widget values are +inlined. Subgraphs are flattened to prefixed node IDs (e.g. `"11:3"`). Defined +in the same schema file; sent via `api.queuePrompt()` in `src/scripts/api.ts`. + +### Where the transform lives today + +`graphToPrompt()` in `src/utils/executionUtil.ts` (lines 26–161) is the +primary conversion function. It runs in this order: + +1. **Virtual node application** — calls `node.applyToGraph()` on all virtual + nodes in execution order, mutating the live graph. +2. **Graph serialization** — `graph.serialize()` produces the workflow JSON, + strips `localized_name`, and calls `compressWidgetInputSlots()`. +3. **Node DTO map** — builds an `ExecutableNodeDTO` map that handles subgraph + flattening; inner node IDs become `"parentId:childId"` prefixes via + `workflowFlattening.ts`. +4. **Prompt assembly** — for each DTO: collects widget values (calling + `widget.serializeValue` when present), resolves input links via + `node.resolveInput()`, and handles the `__type__: 'CURVE'` and + `__value__: array` wrapping conventions. +5. **Dangling link cleanup** — removes inputs referencing nodes not in the + output map. + +Several transforms happen outside `graphToPrompt()` and are not visible to it: + +| Transform | Location | Mechanism | +|---|---|---| +| Dynamic prompt `{a\|b}` substitution | `src/extensions/core/dynamicPrompts.ts` | Overrides `widget.serializeValue` on `nodeCreated` | +| Promoted widget control (`control_after_generate`) | `src/scripts/promotedWidgetControl.ts` | Called from `app.queuePrompt()` before `graphToPrompt` | +| Widget value propagation for UI feedback | `src/extensions/core/widgetValuePropagation.ts` | Separate extension hook, not in prompt path | +| Subgraph input promotion resolution | `src/core/graph/subgraph/resolveConcretePromotedWidget.ts` | Called from within `node.resolveInput()` inside the DTO | + +The extension hook system (`ComfyExtension` in `src/types/comfy.ts`) exposes +`beforeConfigureGraph`, `nodeCreated`, `loadedGraphNode`, and +`afterConfigureGraph` — any extension can inject into the transform at these +points. No contract governs ordering, idempotency, or what state is legal to +mutate. + +### Why this matters + +Any system that reads, modifies, or executes a ComfyUI workflow without the +frontend must independently reproduce every one of these transforms to faithfully +execute the workflow. This affects: + +- **Agent mode** — constructs and submits prompts without user interaction. +- **Hub app mode** — executes workflows on behalf of users from a stored + representation. +- **MCP tools** — invoke workflow execution from external processes. +- **Developer platform / API v2** — the long-stated goal is that the workflow + file is the canonical artifact; the execution payload is derived state. + +Related: FE-1577 (V2 API surface) covers the extension API rather than the +graph transform. Both land on the same consumers. + +## Decision + +**[reconstruction — needs Ben's confirmation]** + +The proposal is additive: no break to the existing workflow format or execution +API. + +1. **Centralize the transform.** All transforms that convert the workflow + representation into the execution payload — virtual node application, link + resolution, subgraph flattening, widget value serialization, dynamic prompt + substitution, promoted widget control — are moved behind a single function + with a documented, stable signature. This replaces the current scattered + call-sites. + +2. **Make transforms replayable from the workflow.** Enough information about + each transform step is encoded in the workflow JSON that the conversion can be + reproduced outside the frontend without the live graph. Specifically: + - The workflow already encodes subgraph structure; flattening must be + derivable from `definitions.subgraphs` alone. + - Dynamic prompt seeds or substitution results are optionally embedded so + reproductions are deterministic. + - Promoted widget values are carried by the host node's serialized state, not + by interior subgraph nodes (consistent with ADR 0009). + +3. **Optionally persist both representations together.** The queue payload may + include both the workflow and the resulting API payload alongside version and + provenance information, so consumers can validate or reproduce the conversion. + The execution payload remains derived state — the workflow is the single + source of truth. + +Mental model: **workflow → explicit named transforms → execution payload**. The +payload is never edited directly; it is always regenerated from the workflow. + +### What this ADR is not deciding + +- The wire format of the API v2 prompt endpoint (FE-1577). +- Whether subgraph definitions should be changed (ADR 0009 governs that). +- How extensions register custom transforms after this centralization — that + registration contract is deferred pending the centralized implementation. + +### Enforceability as a standard for new code + +The specific ask from the 2026-08-21 discussion was whether this can be enforced +as a standard for new code. The answer is: **yes, once the centralized transform +function exists**. At that point: + +- New transform logic must be added to the central pipeline, not to extension + hooks or `queuePrompt()` call-sites. +- The `ComfyExtension.serializeValue` override pattern (currently used by + `dynamicPrompts.ts`) is deprecated in favor of a registered transform step + with explicit ordering and isolation guarantees. +- An ESLint rule or ADR compliance check can flag direct calls to `graph.serialize()` + or `graphToPrompt()` from outside the designated transform module. + +## Consequences + +### Positive + +- External systems (Agent, Hub, MCP, CLI tools) can execute any workflow + faithfully without reimplementing frontend-only transforms. +- The transform pipeline becomes testable in isolation — no live graph required. +- Extension authors get a documented, stable hook rather than relying on + `widget.serializeValue` override or timing-dependent `beforeConfigureGraph`. +- Provenance information in the persisted payload enables future validation, + debugging, and replay. + +### Negative / risks + +- **Migration cost.** `dynamicPrompts.ts`, `promotedWidgetControl.ts`, and any + third-party extension using `serializeValue` overrides must be migrated. The + `serializeValue` override pattern is used by the extension ecosystem (40+ + custom node repos per ADR 0008 amendment). +- **Ordering sensitivity.** The current transforms run in an implicit order + determined by call-site position and extension registration sequence. Making + that order explicit may reveal latent bugs in extensions that rely on it. +- **Reconstruction uncertainty.** This ADR is reconstructed from a single Slack + message. The specific encoding decisions (what exactly goes into the workflow + to make transforms replayable) are not fully specified and must be detailed + before implementation begins. + +## Open questions (requires Ben's input) + +1. Which transforms must be replayable outside the frontend in the initial + scope, and which are deferred? +2. Should the persisted execution payload be in the PNG `pnginfo`, a sidecar + file, or the queue request body only? +3. Is the `serializeValue` deprecation in-scope for this proposal, or is it a + follow-on once the central pipeline stabilizes? +4. Does "encode enough about transforms in the workflow" mean storing transform + output (e.g. resolved dynamic prompt values) or transform parameters (e.g. + random seed used)? From fa6f976a0c95f40416852f515934267a2e549c62 Mon Sep 17 00:00:00 2001 From: GitHub Action Date: Sun, 23 Aug 2026 07:13:58 +0000 Subject: [PATCH 2/3] [automated] Apply ESLint and Oxfmt fixes --- .../0016-workflow-to-execution-payload-transform.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/adr/0016-workflow-to-execution-payload-transform.md b/docs/adr/0016-workflow-to-execution-payload-transform.md index 7a02a204d48..a0aa6530414 100644 --- a/docs/adr/0016-workflow-to-execution-payload-transform.md +++ b/docs/adr/0016-workflow-to-execution-payload-transform.md @@ -54,12 +54,12 @@ primary conversion function. It runs in this order: Several transforms happen outside `graphToPrompt()` and are not visible to it: -| Transform | Location | Mechanism | -|---|---|---| -| Dynamic prompt `{a\|b}` substitution | `src/extensions/core/dynamicPrompts.ts` | Overrides `widget.serializeValue` on `nodeCreated` | -| Promoted widget control (`control_after_generate`) | `src/scripts/promotedWidgetControl.ts` | Called from `app.queuePrompt()` before `graphToPrompt` | -| Widget value propagation for UI feedback | `src/extensions/core/widgetValuePropagation.ts` | Separate extension hook, not in prompt path | -| Subgraph input promotion resolution | `src/core/graph/subgraph/resolveConcretePromotedWidget.ts` | Called from within `node.resolveInput()` inside the DTO | +| Transform | Location | Mechanism | +| -------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------- | +| Dynamic prompt `{a\|b}` substitution | `src/extensions/core/dynamicPrompts.ts` | Overrides `widget.serializeValue` on `nodeCreated` | +| Promoted widget control (`control_after_generate`) | `src/scripts/promotedWidgetControl.ts` | Called from `app.queuePrompt()` before `graphToPrompt` | +| Widget value propagation for UI feedback | `src/extensions/core/widgetValuePropagation.ts` | Separate extension hook, not in prompt path | +| Subgraph input promotion resolution | `src/core/graph/subgraph/resolveConcretePromotedWidget.ts` | Called from within `node.resolveInput()` inside the DTO | The extension hook system (`ComfyExtension` in `src/types/comfy.ts`) exposes `beforeConfigureGraph`, `nodeCreated`, `loadedGraphNode`, and From e53142ed6693ea57dfd3e1d7ead17d5a1f3cc5f4 Mon Sep 17 00:00:00 2001 From: Christian Byrne Date: Sun, 23 Aug 2026 02:41:42 -0700 Subject: [PATCH 3/3] docs(adr): address coderabbit review on ADR 0016 - note transform inventory is non-exhaustive (Easy-Use variables, KJNodes get/set, link rewrites from #15704) and add open question 5 on their scope - caveat replayability: runtime-dependent serializeValue hooks (painter canvas upload, store-held promoted values) need mandatory replay metadata or explicit exclusion - constrain payload pairing to a versioned envelope/sidecar; legacy prompt shape unchanged on the wire - qualify enforceability: call-site restriction immediate, ordering/ idempotency gated on the registration contract; mark section as reconstruction --- ...workflow-to-execution-payload-transform.md | 39 ++++++++++++++++++- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/docs/adr/0016-workflow-to-execution-payload-transform.md b/docs/adr/0016-workflow-to-execution-payload-transform.md index a0aa6530414..5cc0fd9bdc3 100644 --- a/docs/adr/0016-workflow-to-execution-payload-transform.md +++ b/docs/adr/0016-workflow-to-execution-payload-transform.md @@ -97,6 +97,13 @@ API. with a documented, stable signature. This replaces the current scattered call-sites. + This inventory is not exhaustive. #15704 also names ecosystem-level + transforms that rewrite the graph before execution: Easy-Use + "use everywhere" variables, KJNodes get/set nodes, and link rewrites. + Those live in custom-node repos, not this codebase, so whether they enter + the central pipeline (via the deferred registration contract) or remain + outside the standard is open question 5. + 2. **Make transforms replayable from the workflow.** Enough information about each transform step is encoded in the workflow JSON that the conversion can be reproduced outside the frontend without the live graph. Specifically: @@ -107,12 +114,29 @@ API. - Promoted widget values are carried by the host node's serialized state, not by interior subgraph nodes (consistent with ADR 0009). + Caveat: the workflow JSON does not currently contain everything + `graphToPrompt()` consumes. Some `widget.serializeValue` hooks produce + runtime-only data — e.g. the painter widget uploads a canvas and returns a + server-generated file reference, and promoted values can live in + `useWidgetValueStore()` rather than the serialized graph. For each + runtime-dependent transform, replay metadata in the workflow is mandatory, + or the transform must be explicitly listed as non-replayable and excluded + from the replayability guarantee. + 3. **Optionally persist both representations together.** The queue payload may include both the workflow and the resulting API payload alongside version and provenance information, so consumers can validate or reproduce the conversion. The execution payload remains derived state — the workflow is the single source of truth. + Shape constraint: `ComfyApiWorkflow` is a flat dict keyed by node ID, so + `workflow`, `version`, and `provenance` cannot be added as sibling keys + without breaking the payload's validity. Pairing must use a versioned + envelope (or sidecar) around the untouched legacy prompt shape; + `api.queuePrompt()` keeps sending the legacy shape to the existing + endpoint, and consumers of the envelope must be able to reject a derived + payload that is stale relative to its paired workflow. + Mental model: **workflow → explicit named transforms → execution payload**. The payload is never edited directly; it is always regenerated from the workflow. @@ -125,9 +149,17 @@ payload is never edited directly; it is always regenerated from the workflow. ### Enforceability as a standard for new code +**[reconstruction — needs Ben's confirmation]** + The specific ask from the 2026-08-21 discussion was whether this can be enforced -as a standard for new code. The answer is: **yes, once the centralized transform -function exists**. At that point: +as a standard for new code. The reconstructed answer is: **partially at first, +fully once the registration contract exists**. Call-site restriction (new +transform logic goes through the central pipeline, not extension hooks or +`queuePrompt()` call-sites) is enforceable as soon as the centralized function +exists. Ordering, idempotency, and legal-mutation-state guarantees are not +enforceable until the extension registration contract — deferred above — is +defined, along with a migration boundary for existing `serializeValue` users. +At that point: - New transform logic must be added to the central pipeline, not to extension hooks or `queuePrompt()` call-sites. @@ -174,3 +206,6 @@ function exists**. At that point: 4. Does "encode enough about transforms in the workflow" mean storing transform output (e.g. resolved dynamic prompt values) or transform parameters (e.g. random seed used)? +5. Are ecosystem-level graph rewrites (Easy-Use "use everywhere" variables, + KJNodes get/set nodes, link rewrites — see #15704) in scope for the central + pipeline via the registration contract, or explicitly outside the standard?