You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b90a343
Browse filesBrowse the repository at this point in the historyBrowse files
@@ -145,6 +153,7 @@ make test # all tests with -race
145
153
make test-short # unit-only
146
154
make test-count # 3× runs (flake hunt)
147
155
make test-e2e # cluster chaos e2e (~2 min, tag-gated)
156
+
make examples # run every example (each must exit 0 within 60s)
148
157
make cover # HTML coverage
149
158
```
150
159
@@ -165,23 +174,53 @@ Within one scheduler instance, `mu` serializes event handling. Across instances,
165
174
### Sweep on leader acquisition
166
175
When `IsLeader()` flips false→true, the scheduler sweeps all running DAGs and re-enqueues any stranded ready steps. Edge-triggered, not level-triggered. Defaults: 5s poll, 60s sweep timeout. Overlap-guarded.
167
176
168
-
### Breakpoints are computed gates; `Held` and BP fences are independent
169
-
Per-step breakpoints (`BreakBefore`/`BreakAfter` StepOptions, armed via
170
-
`WithActiveBreakpoints` at submit) gate scheduling inside the pure layer:
171
-
`ReadyToRun` refuses a pending step whose before-BP is armed, `depsSatisfied`
172
-
refuses dependents of a done step whose after-BP is armed. The gate is computed
173
-
from immutable config (labels × `DAGMeta.ActiveBreakpoints`) plus the monotonic
174
-
`BPBefore`/`BPAfter = released` flag written only by `ResumeBreakpoint` — the
175
-
persisted `blocked` marks (`bp_before`/`bp_after`/`bp_blocked_at`) are advisory
176
-
observability only, never load-bearing. Resume is debugger "continue": the label
177
-
stays armed; each call releases only what is currently stopped. The pause `Held`
178
-
fence and BP state are separate fields by design — DAG `Resume` and the
179
-
orphan-hold sweep touch only `Held`/`HeldAt`, `ResumeBreakpoint` touches only BP
180
-
state; the two fences compose. Never merge them.
177
+
### Breakpoints are computed gates; `Held`, BP gates and `BPFence` are independent
178
+
Per-step breakpoints are armed by matcher, at submit (`WithArmedBreakpoints`;
179
+
`WithActiveBreakpoints` labels are stored as `BeforeLabel`/`AfterLabel`
180
+
matchers in `DAGMeta.ArmedMatchers`) or at runtime (`ArmBreakpoints` /
181
+
`DisarmBreakpoints` CAS `ArmedMatchers` on meta), and gate scheduling inside the
182
+
pure layer: `ReadyToRun` refuses a pending step whose before-BP is armed,
183
+
`depsSatisfied` refuses dependents of a done step whose after-BP is armed. The
184
+
gate is computed from step identity (step ID, fn name, declared labels) ×
185
+
`ArmedMatchers` (plus the deprecated read-only `ActiveBreakpoints` of pre-v0.9.0
186
+
DAGs) × `BPFence` tokens, plus the monotonic `BPBefore`/`BPAfter = released`
187
+
flag written only by `ResumeBreakpoint`, `ReleaseMatching` and `ReleaseStep`
188
+
(all through `releaseInstance`, the one release path) — the persisted `blocked`
189
+
marks (`bp_before`/`bp_after`/`bp_blocked_at`) are advisory observability only,
190
+
never load-bearing. Resume is debugger "continue": the label stays armed; each
191
+
call releases only what is currently stopped. `ReleaseStep` releases one step
192
+
at one position and never pre-releases a step that has not reached its
193
+
breakpoint (that would silently consume a stop). Disarm never writes
194
+
`released` either, so disarm + re-arm stops a step again.
195
+
196
+
**Runtime arming: meta is the truth, `BPFence` is the dispatch guard.** Meta is
197
+
read through a direct get that may be stale, so a scheduler could miss a fresh
198
+
arm. Two rules close that, and the argument is on `ArmBreakpoints`:
199
+
(1) every dispatcher lists steps first and then reads meta from the stream
200
+
leader (`StateStore.GetMetaLatest`) before dispatching: `enqueueReady` (only
201
+
when something is ready), always in `onResumed` (disarm/release liveness) and
202
+
the sweep, and in `Submit` before the root loop; (2) arm commits meta, then
203
+
CAS-adds `b:<matcherKey>` to each matched pending step and `a:<upstream>` to the
204
+
pending dependents of a matched done step. `persistStatus` refuses
205
+
pending→running while a live token remains (a `b:` token yields to `BPBefore
206
+
released`, an `a:X` token to X's `BPAfter released`). Tokens are removed by
207
+
release (`b:` in the release CAS, `a:X` after X's release), by disarm, and by
208
+
the sweep repair `releaseOrphanedBPFences`; all three judge staleness by
209
+
reading the step first and meta from the leader second, and arm always rewrites
210
+
a token it finds (revision bump) so a concurrent prune loses its CAS. Dynamic
211
+
steps get no tokens: the leader meta read covers them.
212
+
213
+
The pause `Held` fence, the BP gates and `BPFence` are separate by design — DAG
214
+
`Resume` and the orphan-hold sweep touch only `Held`/`HeldAt`; BP releases,
215
+
arm and disarm touch only BP state and `BPFence`; they compose. Never merge
216
+
`Held` and `BPFence`. `StartPaused` uses only the pause fence: meta is created
217
+
paused and every step created `Held` (never create-then-hold, so no sweep sees
218
+
an unheld root), and Submit skips the root-enqueue loop. Workers older than
219
+
v0.10.0 ignore runtime arming: upgrade all of them first.
181
220
182
221
Breakpoint transitions are announced as **informational**`bp_hit`/`bp_resumed`
183
222
events (published best-effort by the advisory-mark CAS winner and by
184
-
`ResumeBreakpoint`; rendered live by `ebctl dag watch`). Schedulers Ack-drop
223
+
`releaseInstance`; rendered live by `ebctl dag watch`). Schedulers Ack-drop
185
224
them — nothing load-bearing may ever depend on their delivery; the durable
186
225
truth is the step record (`dag bp ls`).
187
226
@@ -199,15 +238,25 @@ immutability is load-bearing**: duplicate step enqueues are collapsed by
199
238
msg-id dedupe, which requires every enqueue attempt to resolve the identical
200
239
payload — never add signal-overwrite semantics without redesigning that. A
201
240
`SigRef` arg is NOT a step dependency (distinct `__ebind_signal__` marker;
202
-
excluded from deps/cycles/cascade). The signal gate, the `Held` fence, and the
203
-
BP gates are three independent fences that compose — no release path touches
204
-
another fence's state. Terminal DAGs: `Signal` no-ops without writing;
241
+
excluded from deps/cycles/cascade). The signal gate, the `Held` fence, the
242
+
BP gates and the `BPFence` tokens are four independent fences that compose — no
243
+
release path touches another fence's state. Terminal DAGs: `Signal` no-ops without writing;
205
244
`DeleteDAG` must delete signal records (a re-created pinned `WithDAGID` DAG
206
245
would inherit them). Roots get a signals snapshot at Submit (`dag.go`) since
207
246
they bypass the scheduler gate.
208
247
248
+
### Submit order: validate, hooks, validate options, then claim
249
+
`DAG.Submit` validates structure under `d.mu`, releases it, runs the `SubmitHook`s (they may do I/O; their options go after the caller's), validates the merged options, and only then re-takes the lock and sets `submitted` (re-checking it: two concurrent Submits of one `*DAG`, one wins). A hook or option error therefore persists nothing and leaves the DAG resubmittable. Hooks get a copy of the provisional meta (ID, Labels, DefaultPolicy, CreatedAt). Dynamic steps never pass through Submit; matchers armed by a hook still gate them.
250
+
251
+
### Operator tools open, never create-or-update
252
+
`NewFromNATS`/`NewNatsStore`/`NewNatsBus` use `CreateOrUpdate*`, which rewrites the whole bucket/stream config (replicas included) to whatever the caller passes. Only the deployment that owns the infrastructure (workers) may call them. `ebctl` and any other tool that attaches to a running deployment must use `OpenFromNATS` (`OpenNatsStore`/`OpenNatsBus`), which opens existing resources and errors if they are missing. Before v0.9 ebctl defaulted `--replicas 1` through the create path and silently scaled R=3 deployments to R=1. Regression test: `cmd/ebctl/open_only_test.go`.
253
+
254
+
### Waiting on DAG state: watch, never poll Get
255
+
`StateStore.WatchDAG` streams a DAG's meta + step records (snapshot, `Synced`, then every change). On NatsStore it is a KV `WatchFiltered` (ordered consumer), so it never sees the direct-get staleness below; a closed channel means "re-watch", and the fresh snapshot makes that restart-safe. `WaitForBreakpoint` is built on it and evaluates the computed gate (`ComputeBreakpoints`), never the advisory marks. MemStore's watch never blocks writers: a subscriber whose buffer fills is closed. `WatchDAG` is part of the `StateStore` interface and of `runStoreContract` (`runWatchDAGContract`, also run against NatsStore).
256
+
209
257
### KV reads are direct-get — no read-your-writes
210
-
NATS JetStream KV `Get` is a *direct get*: hardcoded `AllowDirect: true` on the bucket stream, served by any in-sync replica via a queue group, not just the leader. A `Create`/`Update` commits at quorum (leader + 1 of 2 followers); the third replica applies a beat later. So a `Get` issued immediately after a write can be answered by a lagging replica and return a stale value — including `ErrKeyNotFound`/`ErrStepNotFound` for a key that definitely exists. The window is widest right after a node restart (the rejoined replica is back in the direct-get pool but a hair behind). **Never read-back-after-write to recover a revision** — `StateStore.PutStep`/`PutMeta` should return the new revision (as `kv.Create`/`kv.Update` do) and callers CAS from it directly. Writes/CAS are strongly consistent (leader-served, quorum); only this relaxed read path needs care. See `workflow/dag.go::Submit` and the `Step_PutReturnsUsableRevision` store contract.
258
+
NATS JetStream KV `Get` is a *direct get*: hardcoded `AllowDirect: true` on the bucket stream, served by any in-sync replica via a queue group, not just the leader. A `Create`/`Update` commits at quorum (leader + 1 of 2 followers); the third replica applies a beat later. So a `Get` issued immediately after a write can be answered by a lagging replica and return a stale value — including `ErrKeyNotFound`/`ErrStepNotFound` for a key that definitely exists. The window is widest right after a node restart (the rejoined replica is back in the direct-get pool but a hair behind). **Never read-back-after-write to recover a revision** — `StateStore.PutStep`/`PutMeta` should return the new revision (as `kv.Create`/`kv.Update` do) and callers CAS from it directly. When a decision needs the latest meta (runtime
259
+
breakpoints), use `GetMetaLatest`, a leader read. Writes/CAS are strongly consistent (leader-served, quorum); only this relaxed read path needs care. See `workflow/dag.go::Submit` and the `Step_PutReturnsUsableRevision` store contract.
211
260
212
261
### Failed-step error message persistence
213
262
On terminal failure the worker's `StepHook.OnStepFailed` writes both `error_kind` and `error_message` (the handler's `err.Error()`) into the step record, and carries them on the completion event so the scheduler's in-memory `MarkFailed` stays consistent. The message is truncated to `Workflow.MaxStepErrorBytes` (default `DefaultMaxStepErrorBytes` = 4096; **negative ⇒ store kind only**, the compliance opt-out). The step record is the durable, DAG-lifetime source for *why* a step failed (`ebctl dag step get`); the full untruncated `TaskError` also lives in `EBIND_DLQ` (7d) and `EBIND_RESP` (short). The hook persists the record *before* the DLQ publish, so a step showing `failed` always has its error available. Only terminal failures are recorded — mid-retry attempts write nothing (use `worker.Log` middleware to capture interim errors).
0 commit comments