[feat] Add bounded loops and parallel for_each #1079

Activity

1. hsuchifeng

   added a parent issue on Oct 13, 2026

   • Discussion: Workflow Runtime Integration #818
2. hsuchifeng

   marked this as blocked on Oct 13, 2026

   • [feat] Add DAG conditions and composable sub-workflows #1078
3. SaulMoro commented on Oct 13, 2026

   

   SaulMoro

   on Oct 13, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   Two things from the #1097 review:

   • A report is checked against the run's revision. If that lands as is, siblings leased in parallel here will get stale_revision once one of them reports. Checking against the lease's revision would keep them valid.
   • next gives a driver its own open lease first, so the local driver would get the same item back instead of a second one. With max_parallel above 1 it needs some way to ask for another ready item, for example a worker id in the driver identity, or telling next which attempts it's already running.
4. hsuchifeng commented on Oct 14, 2026

   

   hsuchifeng

   on Oct 14, 2026

   CollaboratorAuthor

   More actions

   A report is checked against its own lease revision. One sibling reporting in parallel does not make the others stale_revision.

   next must return another ready item while the same driver already holds a lease. A worker id on the driver identity, or the attempts it is already running, is enough. max_parallel above 1 requires that.

   This stays command-only and blocked only by #1078. It owns command-only for_each, item, and results. Branches, results[].branch, branch-based item after, and session: continue across loop passes move to #1080.
5. SaulMoro commented on Oct 14, 2026

   

   SaulMoro

   on Oct 14, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   Notes from #1078 (#1129) that this issue builds on:

   • Step paths. Child steps are addressed by path (ship.check.classify), and the path grammar already leaves room for [n]. for_each instances can use build[2].implement without reshaping the run. No iteration fields exist yet.
   • Child steps are created only when the workflow step would be the next lease. With one lease at a time, a child runs without the parent changing underneath it. That is also why a ready end-only step can win over an earlier ready step without surprises. Parallel leases remove this guarantee, so child creation and nested end need a fresh look here.
   • Revision. report must carry the run's current revision, and engine-side settling (skips, if, end, child expansion) happens inside next, before a lease is handed out, without bumping it. With several open leases, one report bumps the revision under another lease. This is the "report revision" point in my earlier comment.
   • Readiness. A step is ready only when everything it reads has settled: its needs, plus every step its if, templates and with read (canRun). for_each: steps.plan.outputs.issues should go through the same check. A step that runs an inline workflow also waits for every outer step its inline steps read; without that, an outer read that is skipped early leaves the inline block waiting forever. Parallel leases need the same rule.
   • needs is required after the first step (needs: [] declares another root). loop.back_to re-pends steps that already settled, so the skip rules (a stopped dependency skips a step unless its if reads it, and a join is skipped only when all its dependencies were skipped) must be re-evaluated on each pass.

   The engine rules that span files are in docs/dev/workflow-engine.md, added by #1129.
6. SaulMoro commented on Oct 15, 2026

   

   SaulMoro

   on Oct 15, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   Notes from checking this against #818, the scorer label agreed in #1085, and #1080:

   • Dependents wait until the loop settles. Discussion: Workflow Runtime Integration #818 says so ("Dependents wait until the loop settles and see its last output"), and implement-task depends on it. done needs only e2e. When e2e fails on the first pass, done becomes ready, its if is false, and it is skipped. fix loops back, e2e passes, and done stays skipped, so the workflow never reaches its done end. A step that needs a step inside a back_to range could wait until the looping step settles.
   • A pass needs an id. [feat] Add workflow scorers and recorded quality labels #1085 identifies a scored step by { runId, step }. In review-and-fix, review runs up to three times under one path. Sampling then picks all passes or none, and a later pass replaces the label of an earlier one. Branch names in [feat] Add workflow workspaces, branches, and session continuation #1080 hit the same collision across rounds. If this issue gives each pass a number that the run keeps (review#2, or a pass on the attempt), [feat] Add workflow scorers and recorded quality labels #1085 and [feat] Add workflow workspaces, branches, and session continuation #1080 can both use it. The label record is already locked in [feat] Add workflow scorers and recorded quality labels #1085, so this would reopen it, either with a pass field or with a step that includes the pass. That is cheaper before any label is published.
   • Worker ids across a restart. Of the two options agreed above, a worker id works only if it survives a restart. next returns a driver its own open lease by comparing kind, machine and workspace (sameDriver), which is how a run continues after the laptop closes. A worker id generated per process would join that identity, so a restarted driver would never get its leases back. A restart case in the acceptance criteria would cover either option.
   • Reading the previous pass. In plan-change, plan reads steps.decide.note, and decide comes after plan. Pull accepts reads of needs ancestors only (parse.ts), and a step waits for every step it reads, so the planner would wait for a decide that waits for the planner. A loop needs a defined way to read the previous pass. The test from Discussion: Workflow Runtime Integration #818: after revise with a note, the next planner pass sees the note and runs at once, and decide asks again. With [feat] Add manual run controls and human gates #1081 adding note, whichever lands second owns this case. Clearing choice and note when back_to re-pends decide would lose the note.
   • Ends with parallel leases. Discussion: Workflow Runtime Integration #818 says reaching an end skips pending steps and cancels open leases. The engine only skips pending steps (engine.ts:420), which was enough with one lease at a time. With parallel items, a sibling can hold a lease when an end is reached. Its later report should be rejected. An end inside a child workflow should cancel only that child's leases, not the parent's.
   • Smaller: an item whose after item failed is skipped (Discussion: Workflow Runtime Integration #818). That is not in the acceptance criteria.
7. SaulMoro

   self-assigned this

   on Oct 15, 2026
8. SaulMoro commented on Oct 15, 2026

   

   SaulMoro

   on Oct 15, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   Plan for this issue, before the PR. It lands as one PR. Points that touch #1081, #1085 or #1080 are marked.

   • next takes the attempts a driver is running. The payload gains optional running: [attempts]. A driver's own open lease that is not in that list is returned before any new work, so a restarted driver sends running: [] and gets its leases back one by one. Otherwise next leases the next ready step. canTakeLease: false makes a call that only settles the run and answers done or wait. A driver that never sends running behaves as today. No worker id, so the identity in sameDriver does not change.
   • Any ready step can run in parallel, not only for_each items. Independent needs: [] branches can be leased together when the driver asks with running. Steps are still found, and workflow steps still started, in file order. Another driver still gets wait, as multi-machine leases are out of scope.
   • A report is checked against its own lease revision. The answers for unknown attempts stay as today (stale_revision on an old revision, otherwise wrong_attempt).
   • New step status cancelled ([feat] Add manual run controls and human gates #1081). When an end or break is reached, open leases in that scope become cancelled. Their later report returns lease_cancelled, also after a loop has re-run that scope. Every next response lists, in cancelled, the attempts from running that are no longer leased, so a driver can stop those processes. Pending steps are still skipped. An end inside a child workflow or an item cancels only that scope. When parallel reports reach two ends before the next next, the one reached first wins. [feat] Add manual run controls and human gates #1081 can reuse the status for manual cancel.
   • Dependents wait until a loop settles. A step outside a back_to range that needs or reads a step inside it is not evaluated, for skip or for readiness, until the looping step is final. That covers done in implement-task. The looping step runs at most max_iterations times. With while, the loop runs out when while is still true after the last run. Without while, the range goes back once more, and the step fails loop_exhausted when it would run again after its if. So in implement-task the e2e gate checks the third fix, and in review-and-fix a passing fourth review ends the loop normally.
   • One step loop per range. Pull rejects step loops whose ranges share a step, and a step loop in a workflow that has its own loop. Loops nest through a sub-workflow, as every example in Discussion: Workflow Runtime Integration #818 does.
   • Each pass has a number ([feat] Add workflow scorers and recorded quality labels #1085, [feat] Add workflow workspaces, branches, and session continuation #1080). A step keeps pass and its passes, and the lease carries pass. When a loop re-runs a child workflow or a fan-out, the new instances continue the numbering of their path, and the old ones move to the run's archive. So { runId, step, pass } is unique in a run, across rounds and restarts. Expressions read steps.<id>.passes, every pass of the step including the current one once it settles.
   • Reading the previous pass ([feat] Add manual run controls and human gates #1081). Inside a loop range, a step can read a later step of the same range, such as plan reading steps.decide in a back_to: plan loop. The read gets that step's last archived pass, null on the first pass, and does not wait for it. [feat] Add manual run controls and human gates #1081 only needs to add choice and note to what a pass keeps.
   • Workflow loops and break. Any end reached in a round stops the loop with that end's outcome. A failed step alone does not, so ready-issues plans again after a failed item. break: <expr> ends the workflow, or its loop, as success when the expression is true after its step succeeds. It is not allowed inside a back_to range, where if and while already end the loop.
   • for_each. Items expand to build[0], build[1], and so on, 0-based, when the step is reached. {{ item }} passes the whole item. after uses item ids. An item whose after item failed, was skipped or was cancelled is skipped. A cycle fails the step and names it. max_parallel caps started items. results is [{ item, outcome, outputs }] with no branch fields.
   • The local driver runs in parallel with workflows.max_parallel in the local config, 1 by default. Prompt steps still run alone, because the read-only check looks at the whole worktree. With more than 1, commands share the working tree and run without a terminal, and Ctrl-C stops all of them.
   • Expressions grow (Discussion: Workflow Runtime Integration #818). A path can take an index, such as steps.build.results[2].outcome. Expressions add <, <=, > and >= on numbers, and length, any, all and filter over arrays, with predicates relative to the element as in jq, such as any(steps.build.results, .outputs.changed). So for_each: filter(steps.build.results, .outcome == 'failed') runs only the failed items again. A {{ }} template still reads one path. This extends the operators listed in Discussion: Workflow Runtime Integration #818.

   Out of scope here: branches, results[].branch, session: continue (#1080), choose and note (#1081), nested step loops in one workflow, negative indexes, slices and arithmetic in expressions.
9. SaulMoro commented on Oct 15, 2026

   

   SaulMoro

   on Oct 15, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   What changed from the plan above, now that #1148 implements it:

   • End order. "The end reached first wins" holds whether or not the driver calls next between reports. Before it accepts a report, report settles the events that came before it.
   • Run records ([feat] Publish privacy-preserving workflow run records and stats #1083). Every entry has its pass, and a step that ran more than once has one entry per pass, keyed by { run, step, pass }, as [feat] Add workflow scorers and recorded quality labels #1085 and [feat] Add workflow workspaces, branches, and session continuation #1080 expect. stepCounts gains cancelled, and a cancelled step's entry has outcome: 'cancelled'. A run that failed without reaching an end records failure: { code }, without the message, and only for codes teamai sets. tool and model are kept only when they are a local driver tool and a name-shaped model. next on a settled run answers record only if that run's end record was queued.
   • Templates. A {{ }} reads one path, or now evaluates an expression as an if does, such as {{ length(filter(steps.build.results, .outcome == 'failed')) }}. pull checks it like an if and reports a mistaken one, such as {{ lenght(steps.build.results) }}. Text in other template languages, such as Vue's {{ filter(items) }}, Jinja or ${{ }}, still stays as written, and so does {{ item }} outside a for_each step. Expressions nest at most 100 levels.
   • Performance. With --driver local, an uncapped for_each of 1,000 items takes about 42 s and max_parallel: 8 over 200 items about 3.5 s. Most of the 42 s is engine overhead, because every call settles each running item and parses the whole run file. perf(workflow): engine overhead dominates a long uncapped for_each #1162 tracks this.

   The out-of-scope list in the plan is unchanged.
10. jeff-r2026

    closed this as completedin #1148on Oct 16, 2026