Skip to content

Commit b90a343

Browse files
authored
Merge pull request #35 from F1bonacc1/fix/ebctl-open-only
fix(ebctl): open the workflow bucket and stream without reconfiguring them
2 parents a2a6805 + b0ae7ae commit b90a343

54 files changed

Lines changed: 6724 additions & 231 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,9 @@ jobs:
9595
- name: Verify ebctl --help
9696
run: ./bin/ebctl --help
9797

98+
- name: Run examples
99+
run: make examples
100+
98101
vuln:
99102
name: govulncheck
100103
runs-on: ubuntu-latest

‎CLAUDE.md‎

Lines changed: 72 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -88,20 +88,23 @@ workflow/
8888
errors.go ErrStepFailed, ErrStepSkipped sentinels
8989
ref.go Ref type + ResolveArgs (pure substitution)
9090
state.go DAGState + MarkDone/MarkFailed/MarkSkipped (pure)
91-
store.go StateStore interface
91+
store.go StateStore interface (GetMetaLatest: leader read) + DAGChange (WatchDAG stream item)
9292
store_mem.go In-memory impl (tests + exported for external use)
9393
store_nats.go JetStream KV impl
9494
events.go EventBus interface + Event types
9595
events_mem.go Channel-based impl for tests
9696
events_nats.go JetStream events impl
9797
enqueuer.go Enqueuer + LeaderElector interfaces
9898
enqueuer_nats.go JetStream task enqueuer
99-
workflow.go Workflow coordinator struct
99+
workflow.go Workflow coordinator struct; SubmitHook + UseSubmitHook
100100
nats.go NewFromNATS convenience constructor
101101
step.go Step + StepOption
102102
dag.go DAG builder, Submit, cycle detection
103103
scheduler.go handleEvent + watchLeadership + sweep
104-
breakpoint.go per-step breakpoints: ComputeBreakpoints, ListBreakpoints, ResumeBreakpoint
104+
breakpoint.go per-step breakpoints: ComputeBreakpoints, ListBreakpoints, ResumeBreakpoint, ReleaseStep
105+
breakpoint_runtime.go ArmBreakpoints/DisarmBreakpoints (runtime arming) + BPFence token helpers + sweep repair
106+
wait_breakpoint.go WaitForBreakpoint: fold a WatchDAG stream, return the first matching stop
107+
matcher.go BPMatcher (position × step|fn|label pattern, path.Match globs) + Key/Validate/Matches (pure); gate use in state.go bpArmed
105108
signal.go external signals: Signal (deliver), ComputeSignals/ListSignalInfo (observability)
106109
hook.go workflow.StepHook (implements worker.StepHook); persists error_kind+message on failure
107110
context.go FromContext — dynamic step addition
@@ -116,7 +119,10 @@ cmd/demo/
116119
117120
cmd/ebctl/ operator CLI (cobra)
118121
main.go entrypoint
119-
internal/cli/ shared Context (NATS conn, Workflow, Printer)
122+
internal/cli/ shared Context (NATS conn, Workflow, Printer); opens the
123+
workflow via OpenFromNATS (open-only, never reconfigures);
124+
matcher.go: shared --before|--after + --step|--fn|--label flags
125+
(dag bp ls/wait/continue/release/resume/arm/disarm)
120126
internal/commands/dag dag ls/get/tree/step (get|result)/watch/cancel/rm
121127
internal/commands/dlq dlq ls/show/watch/requeue/purge
122128
internal/commands/stream stream ls/info/consumer/peek/purge/rmmsg
@@ -127,10 +133,12 @@ cmd/ebctl/ operator CLI (cobra)
127133

128134
- **Pure unit tests** — no NATS, no goroutines beyond test runner:
129135
- `task/retry_test.go`
130-
- `workflow/state_test.go`, `workflow/ref_test.go`, `workflow/dag_test.go`
136+
- `workflow/state_test.go`, `workflow/ref_test.go`, `workflow/dag_test.go`, `workflow/matcher_test.go`
131137
- **Unit with fakes** — MemStore + MemBus + captureEnq:
132138
- `workflow/scheduler_test.go`, `workflow/store_test.go`
133139
- `workflow/breakpoint_test.go` (pure gate-predicate tests + fakes-driven scheduler/resume tests)
140+
- `workflow/release_step_test.go`, `workflow/start_paused_test.go`, `workflow/wait_breakpoint_test.go`, `workflow/matcher_gate_test.go`, `workflow/submit_hook_test.go`
141+
- `workflow/breakpoint_runtime_test.go` (arm/disarm, fence tokens, stale-meta stores, sweep repair), `workflow/breakpoint_race_test.go` (arm vs. completion, 500 interleavings)
134142
- `workflow/signal_test.go` (pure gate/ResolveArgs tests + fakes-driven Signal/scheduler tests)
135143
- **Integration** — real in-process NATS via `embed.StartNode`:
136144
- `worker/worker_test.go`, `workflow/integration_test.go`, `workflow/breakpoint_integration_test.go`, `workflow/signal_integration_test.go`
@@ -145,6 +153,7 @@ make test # all tests with -race
145153
make test-short # unit-only
146154
make test-count # 3× runs (flake hunt)
147155
make test-e2e # cluster chaos e2e (~2 min, tag-gated)
156+
make examples # run every example (each must exit 0 within 60s)
148157
make cover # HTML coverage
149158
```
150159

@@ -165,23 +174,53 @@ Within one scheduler instance, `mu` serializes event handling. Across instances,
165174
### Sweep on leader acquisition
166175
When `IsLeader()` flips false→true, the scheduler sweeps all running DAGs and re-enqueues any stranded ready steps. Edge-triggered, not level-triggered. Defaults: 5s poll, 60s sweep timeout. Overlap-guarded.
167176

168-
### Breakpoints are computed gates; `Held` and BP fences are independent
169-
Per-step breakpoints (`BreakBefore`/`BreakAfter` StepOptions, armed via
170-
`WithActiveBreakpoints` at submit) gate scheduling inside the pure layer:
171-
`ReadyToRun` refuses a pending step whose before-BP is armed, `depsSatisfied`
172-
refuses dependents of a done step whose after-BP is armed. The gate is computed
173-
from immutable config (labels × `DAGMeta.ActiveBreakpoints`) plus the monotonic
174-
`BPBefore`/`BPAfter = released` flag written only by `ResumeBreakpoint` — the
175-
persisted `blocked` marks (`bp_before`/`bp_after`/`bp_blocked_at`) are advisory
176-
observability only, never load-bearing. Resume is debugger "continue": the label
177-
stays armed; each call releases only what is currently stopped. The pause `Held`
178-
fence and BP state are separate fields by design — DAG `Resume` and the
179-
orphan-hold sweep touch only `Held`/`HeldAt`, `ResumeBreakpoint` touches only BP
180-
state; the two fences compose. Never merge them.
177+
### Breakpoints are computed gates; `Held`, BP gates and `BPFence` are independent
178+
Per-step breakpoints are armed by matcher, at submit (`WithArmedBreakpoints`;
179+
`WithActiveBreakpoints` labels are stored as `BeforeLabel`/`AfterLabel`
180+
matchers in `DAGMeta.ArmedMatchers`) or at runtime (`ArmBreakpoints` /
181+
`DisarmBreakpoints` CAS `ArmedMatchers` on meta), and gate scheduling inside the
182+
pure layer: `ReadyToRun` refuses a pending step whose before-BP is armed,
183+
`depsSatisfied` refuses dependents of a done step whose after-BP is armed. The
184+
gate is computed from step identity (step ID, fn name, declared labels) ×
185+
`ArmedMatchers` (plus the deprecated read-only `ActiveBreakpoints` of pre-v0.9.0
186+
DAGs) × `BPFence` tokens, plus the monotonic `BPBefore`/`BPAfter = released`
187+
flag written only by `ResumeBreakpoint`, `ReleaseMatching` and `ReleaseStep`
188+
(all through `releaseInstance`, the one release path) — the persisted `blocked`
189+
marks (`bp_before`/`bp_after`/`bp_blocked_at`) are advisory observability only,
190+
never load-bearing. Resume is debugger "continue": the label stays armed; each
191+
call releases only what is currently stopped. `ReleaseStep` releases one step
192+
at one position and never pre-releases a step that has not reached its
193+
breakpoint (that would silently consume a stop). Disarm never writes
194+
`released` either, so disarm + re-arm stops a step again.
195+
196+
**Runtime arming: meta is the truth, `BPFence` is the dispatch guard.** Meta is
197+
read through a direct get that may be stale, so a scheduler could miss a fresh
198+
arm. Two rules close that, and the argument is on `ArmBreakpoints`:
199+
(1) every dispatcher lists steps first and then reads meta from the stream
200+
leader (`StateStore.GetMetaLatest`) before dispatching: `enqueueReady` (only
201+
when something is ready), always in `onResumed` (disarm/release liveness) and
202+
the sweep, and in `Submit` before the root loop; (2) arm commits meta, then
203+
CAS-adds `b:<matcherKey>` to each matched pending step and `a:<upstream>` to the
204+
pending dependents of a matched done step. `persistStatus` refuses
205+
pending→running while a live token remains (a `b:` token yields to `BPBefore
206+
released`, an `a:X` token to X's `BPAfter released`). Tokens are removed by
207+
release (`b:` in the release CAS, `a:X` after X's release), by disarm, and by
208+
the sweep repair `releaseOrphanedBPFences`; all three judge staleness by
209+
reading the step first and meta from the leader second, and arm always rewrites
210+
a token it finds (revision bump) so a concurrent prune loses its CAS. Dynamic
211+
steps get no tokens: the leader meta read covers them.
212+
213+
The pause `Held` fence, the BP gates and `BPFence` are separate by design — DAG
214+
`Resume` and the orphan-hold sweep touch only `Held`/`HeldAt`; BP releases,
215+
arm and disarm touch only BP state and `BPFence`; they compose. Never merge
216+
`Held` and `BPFence`. `StartPaused` uses only the pause fence: meta is created
217+
paused and every step created `Held` (never create-then-hold, so no sweep sees
218+
an unheld root), and Submit skips the root-enqueue loop. Workers older than
219+
v0.10.0 ignore runtime arming: upgrade all of them first.
181220

182221
Breakpoint transitions are announced as **informational** `bp_hit`/`bp_resumed`
183222
events (published best-effort by the advisory-mark CAS winner and by
184-
`ResumeBreakpoint`; rendered live by `ebctl dag watch`). Schedulers Ack-drop
223+
`releaseInstance`; rendered live by `ebctl dag watch`). Schedulers Ack-drop
185224
them — nothing load-bearing may ever depend on their delivery; the durable
186225
truth is the step record (`dag bp ls`).
187226

@@ -199,15 +238,25 @@ immutability is load-bearing**: duplicate step enqueues are collapsed by
199238
msg-id dedupe, which requires every enqueue attempt to resolve the identical
200239
payload — never add signal-overwrite semantics without redesigning that. A
201240
`SigRef` arg is NOT a step dependency (distinct `__ebind_signal__` marker;
202-
excluded from deps/cycles/cascade). The signal gate, the `Held` fence, and the
203-
BP gates are three independent fences that compose — no release path touches
204-
another fence's state. Terminal DAGs: `Signal` no-ops without writing;
241+
excluded from deps/cycles/cascade). The signal gate, the `Held` fence, the
242+
BP gates and the `BPFence` tokens are four independent fences that compose — no
243+
release path touches another fence's state. Terminal DAGs: `Signal` no-ops without writing;
205244
`DeleteDAG` must delete signal records (a re-created pinned `WithDAGID` DAG
206245
would inherit them). Roots get a signals snapshot at Submit (`dag.go`) since
207246
they bypass the scheduler gate.
208247

248+
### Submit order: validate, hooks, validate options, then claim
249+
`DAG.Submit` validates structure under `d.mu`, releases it, runs the `SubmitHook`s (they may do I/O; their options go after the caller's), validates the merged options, and only then re-takes the lock and sets `submitted` (re-checking it: two concurrent Submits of one `*DAG`, one wins). A hook or option error therefore persists nothing and leaves the DAG resubmittable. Hooks get a copy of the provisional meta (ID, Labels, DefaultPolicy, CreatedAt). Dynamic steps never pass through Submit; matchers armed by a hook still gate them.
250+
251+
### Operator tools open, never create-or-update
252+
`NewFromNATS`/`NewNatsStore`/`NewNatsBus` use `CreateOrUpdate*`, which rewrites the whole bucket/stream config (replicas included) to whatever the caller passes. Only the deployment that owns the infrastructure (workers) may call them. `ebctl` and any other tool that attaches to a running deployment must use `OpenFromNATS` (`OpenNatsStore`/`OpenNatsBus`), which opens existing resources and errors if they are missing. Before v0.9 ebctl defaulted `--replicas 1` through the create path and silently scaled R=3 deployments to R=1. Regression test: `cmd/ebctl/open_only_test.go`.
253+
254+
### Waiting on DAG state: watch, never poll Get
255+
`StateStore.WatchDAG` streams a DAG's meta + step records (snapshot, `Synced`, then every change). On NatsStore it is a KV `WatchFiltered` (ordered consumer), so it never sees the direct-get staleness below; a closed channel means "re-watch", and the fresh snapshot makes that restart-safe. `WaitForBreakpoint` is built on it and evaluates the computed gate (`ComputeBreakpoints`), never the advisory marks. MemStore's watch never blocks writers: a subscriber whose buffer fills is closed. `WatchDAG` is part of the `StateStore` interface and of `runStoreContract` (`runWatchDAGContract`, also run against NatsStore).
256+
209257
### KV reads are direct-get — no read-your-writes
210-
NATS JetStream KV `Get` is a *direct get*: hardcoded `AllowDirect: true` on the bucket stream, served by any in-sync replica via a queue group, not just the leader. A `Create`/`Update` commits at quorum (leader + 1 of 2 followers); the third replica applies a beat later. So a `Get` issued immediately after a write can be answered by a lagging replica and return a stale value — including `ErrKeyNotFound`/`ErrStepNotFound` for a key that definitely exists. The window is widest right after a node restart (the rejoined replica is back in the direct-get pool but a hair behind). **Never read-back-after-write to recover a revision** — `StateStore.PutStep`/`PutMeta` should return the new revision (as `kv.Create`/`kv.Update` do) and callers CAS from it directly. Writes/CAS are strongly consistent (leader-served, quorum); only this relaxed read path needs care. See `workflow/dag.go::Submit` and the `Step_PutReturnsUsableRevision` store contract.
258+
NATS JetStream KV `Get` is a *direct get*: hardcoded `AllowDirect: true` on the bucket stream, served by any in-sync replica via a queue group, not just the leader. A `Create`/`Update` commits at quorum (leader + 1 of 2 followers); the third replica applies a beat later. So a `Get` issued immediately after a write can be answered by a lagging replica and return a stale value — including `ErrKeyNotFound`/`ErrStepNotFound` for a key that definitely exists. The window is widest right after a node restart (the rejoined replica is back in the direct-get pool but a hair behind). **Never read-back-after-write to recover a revision** — `StateStore.PutStep`/`PutMeta` should return the new revision (as `kv.Create`/`kv.Update` do) and callers CAS from it directly. When a decision needs the latest meta (runtime
259+
breakpoints), use `GetMetaLatest`, a leader read. Writes/CAS are strongly consistent (leader-served, quorum); only this relaxed read path needs care. See `workflow/dag.go::Submit` and the `Step_PutReturnsUsableRevision` store contract.
211260

212261
### Failed-step error message persistence
213262
On terminal failure the worker's `StepHook.OnStepFailed` writes both `error_kind` and `error_message` (the handler's `err.Error()`) into the step record, and carries them on the completion event so the scheduler's in-memory `MarkFailed` stays consistent. The message is truncated to `Workflow.MaxStepErrorBytes` (default `DefaultMaxStepErrorBytes` = 4096; **negative ⇒ store kind only**, the compliance opt-out). The step record is the durable, DAG-lifetime source for *why* a step failed (`ebctl dag step get`); the full untruncated `TaskError` also lives in `EBIND_DLQ` (7d) and `EBIND_RESP` (short). The hook persists the record *before* the DLQ publish, so a step showing `failed` always has its error available. Only terminal failures are recorded — mid-retry attempts write nothing (use `worker.Log` middleware to capture interim errors).

‎Makefile‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,10 @@ E2E_TIMEOUT ?= 600s
4141
test-e2e: ## Run the cluster chaos e2e test (3-node embedded NATS; E2E_COUNT=n to repeat)
4242
$(GO) test -race -tags e2e -count=$(E2E_COUNT) -timeout $(E2E_TIMEOUT) -v ./e2e/
4343

44+
.PHONY: examples
45+
examples: ## Run every example; each must exit 0 (EXAMPLE_TIMEOUT=60s each)
46+
GO=$(GO) ./examples/run-all.sh
47+
4448
.PHONY: bench
4549
bench: ## Run benchmarks
4650
$(GO) test -run=- -bench=. -benchmem $(PKGS)

0 commit comments

Comments
 (0)