Skip to content

Commit f069a98

Browse files
Brian Arnoldclaude
andcommitted
docs: log O5 — reductions persist results from incomplete member sets (NPIPE-204)
After an upstream pod's output changes, the first job.run() emits only the rows recomputed in that pass. For a per-row pod that emission is complete; for a many->one operator it is partial — the group is reduced over only the members present, the reducing pod executes, and its result is persisted with nothing marking it incomplete. It self-corrects on the next run. Measured with two upstream pod stages: the per-row control converges in one run while the grouped path needs two, executing on a one-member group first. So the behaviour is NOT identical with and without GroupBy, which is how it was previously characterised. Batch behaves the same way, so this is pre-existing pipeline semantics surfaced by reduction rather than something GroupBy introduced — but it matters more for GroupBy, which exists precisely so a pod can reason over a complete group. Not fixable inside the operator: an operator sees whatever its upstream emits and cannot know whether a group is complete. Logged for reviewer judgment rather than fixed here. NPIPE-204 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 2220a31 commit f069a98

1 file changed

Lines changed: 51 additions & 0 deletions

File tree

DESIGN_ISSUES.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -789,6 +789,57 @@ place. Tag uniqueness is assumed throughout the operator layer and enforced nowh
789789
Fix: enforce tag uniqueness at stream construction (raising `DuplicateTagError`), which makes
790790
this branch unreachable with more than one member.
791791

792+
---
793+
### O5 — A reduction persists a result computed from an incomplete member set
794+
**Status:** open
795+
**Severity:** high
796+
797+
When an upstream pod's output changes, the first `job.run()` afterwards emits only the rows
798+
that were recomputed in that pass. For a per-row pod that emission is *complete* — each row is
799+
independent, so processing just the changed one is correct. For a many→one operator it is
800+
*partial*: the group is reduced over only the members present in that pass, the reducing pod
801+
executes, and its result is persisted with nothing marking it incomplete. The next run emits
802+
the full set and the group is recomputed correctly.
803+
804+
Measured with two upstream pod stages (`source → sync_like → stringify → [GroupBy] → pod`),
805+
one member of one group changed, same Delta store, fresh objects per run:
806+
807+
```
808+
CONTROL (per-row, no GroupBy)
809+
changed run1 sync_like(99), stringify(sync_99), pod(sync_99.parquet) <- converged
810+
changed run2 []
811+
812+
GROUPED
813+
changed run1 sync_like(99), stringify(sync_99), GROUP ['sync_99'] <- ONE member
814+
changed run2 GROUP ['sync_99', 'sync_1'] <- correct
815+
changed run3 []
816+
```
817+
818+
The control converges in a single run; only the grouped path needs a second one, and it
819+
executes on an incomplete set first.
820+
821+
Consequence: a reducing pod with a side effect writes a complete-looking artifact from partial
822+
input. For the motivating consumer (`common_clock_op`) an intermediate run can produce an
823+
`alignment.json` built from one of a session's probes. It is replaced on the next run, so a
824+
driver that runs to convergence never observes it — but a consumer reading between runs gets a
825+
wrong answer with no signal.
826+
827+
`Batch` behaves identically, so this is pre-existing pipeline semantics surfaced by reduction
828+
rather than something `GroupBy` introduced. It is more consequential for `GroupBy`, because
829+
`GroupBy` exists specifically so a pod can reason over a *complete* group, whereas nobody
830+
derives a result from `Batch` membership.
831+
832+
Not fixable inside the operator: an operator sees whatever its upstream emits and has no way to
833+
know whether a group is complete. A fix needs a completeness signal at the node level — either
834+
the upstream emitting its full cached set on every pass, or a reducing node deferring execution
835+
until its inputs are known settled.
836+
837+
History: first reported as a cache-corruption bug (wrong group recomputing, tag/data
838+
misalignment), then retracted as a propagation lag "identical with and without `GroupBy`". Both
839+
framings are wrong. There is no tag/data misalignment and no cache defect — but the control
840+
above shows the behaviour is *not* identical, because a partial emission is harmless per-row
841+
and harmful for a reduction.
842+
792843
---
793844

794845
## `src/orcapod/core/` — AddResult pod and Pod Groups

0 commit comments

Comments
 (0)