Commit 1378f7a
feat(runner): error handling foundation and Tier 2 fatal funnel
Combined error-handling P0 implementation per `error_handling_design.md`
(included). Squashes the foundation (PR #82) and storage funnel (PR #83)
work into a single commit on top of `main` (which now carries the
RuntimeCpnet, snapshot-debounce, hibernate_after, $callers, and async
RepoCase changes from #76 / #77 / #79 / #80).
## Errors facade and exception vocabulary
- New `ColouredFlow.Runner.Errors` central classifier: `tier/1`,
`error_code/1`, `lifecycle?/1`, `to_persisted_reason/1`,
`build_exception/2`. Documented as best-effort static dispatch — for
context-sensitive classification subscribers should rely on the
telemetry metadata emitted at the call site.
- Stable `error_code` field on existing public exceptions
(`NonLiveWorkitem`, `InvalidWorkitemTransition`, `UnboundActionOutput`,
`UnsufficientTokensToConsume`, `ColourSetMismatch`,
`Expression.InvalidResult`).
- New caller-facing exceptions: `EnactmentNotRunning`,
`EnactmentTimeout`, `EnactmentCallFailed`, `StoragePersistenceFailed`.
- New Tier 2 enactment-fatal exceptions: `StateDrift`, `SnapshotCorrupt`,
`ReplayFailed`, `EnactmentDataMissing`, `CpnetCorrupt`.
- Extend `Runner.Exception.__reasons__/0` (`Ecto.Enum` for
`enactment_logs.exception.reason`) with the five new fatal reasons.
## Caller-safe wrapper
- `WorkitemTransition.call_enactment/3` translates the full
`GenServer.call/3` `:exit` surface (`:noproc`, `:timeout`, `:shutdown`,
`:normal`, `:nodedown`, `:killed`, `:calling_self`, catch-all) into
typed exceptions returned via `{:error, exception}`.
- `Enactment.Supervisor.terminate_enactment/2` and `Runner.terminate_enactment/2`
reuse the same wrapper; return type widened from `:ok` to
`:ok | {:error, Exception.t()}` (documented breaking change in
`Runner` `@moduledoc`).
- `Registry.whereis/1` for explicit pid lookup.
## Tier 2 fatal-stop funnel
- New private `Enactment.to_exception/3`: persists the enactment as
`:exception` via `Storage.exception_occurs/3`, emits a lifecycle
`:exception` telemetry event with full metadata (`tier`, `lifecycle`,
`severity`, `source_phase`, `exception_reason`, `error_code`,
`exception`, `degraded`), and stops the GenServer with
`{:shutdown, {:fatal, reason}}` so supervisor restart intensity is
not consumed.
- The existing termination-criteria fatal path is folded into the
funnel.
- New `perform_termination/4` for explicit / implicit / force
termination success path; emits structured `{:shutdown, {:terminated, type}}`.
- Caller storage failures from `start` / `complete` / `withdraw` /
`produce` / `terminate` route through the funnel as `:state_drift`
(synthesised `StateDrift` exception threaded back to the caller).
- Bootstrap and async snapshot write failures log and continue (not
fatal); the async snapshot path layers persistence-error handling on
top of the existing mailbox-drain debounce semantics.
- `terminate/2` recognises `{:shutdown, {:fatal, _}}`,
`{:shutdown, {:terminated, _}}`, `{:fatal_persistence_failed, _}` and
emits stable `:stop` event metadata (no more `"unknown"` placeholder).
## Cross-callback rescue
- `populate_state` boot is split into `load_initial_snapshot/1`,
`replay_occurrences/2`, `maybe_persist_bootstrap_snapshot/2` so each
rescue scope attributes errors precisely: snapshot decode ->
`:snapshot_corrupt`, occurrence replay -> `:replay_failed`, missing
enactment row -> `:enactment_data_missing`.
- Both `:calibrate_workitems` `handle_continue` clauses now rescue
flow-decode value-shape errors -> `:cpnet_corrupt` and
`Ecto.NoResultsError` -> `:enactment_data_missing`.
- `handle_call({:complete_workitems, _}, _, _)` likewise rescues missing
/ corrupt flow errors at the callback boundary, replying with the
matching typed exception before the GenServer stops.
- `catchup_snapshot/2` now returns `{snapshot, replayed_steps}`;
populate_state emits `resumed: true | false` and `replayed_steps`
metadata on the `:start` lifecycle event so subscribers can
distinguish a fresh boot from a crash recovery without breaking
existing `[:start]` listeners.
## Storage contract
- Storage callbacks that previously raised on `Multi` rollback or
unexpected row counts now return labelled error tuples
(`{:error, {reason, ctx}}`). Affected: `exception_occurs/3`,
`insert_enactment/1`, `terminate_enactment/4`, `produce_workitems/2`,
`start_workitems/2`, `withdraw_workitems/2`, `complete_workitems/4`,
`take_enactment_snapshot/2`.
- `Storage.Default` switched to `Ecto.Multi.run` + `Repo.get/2` for the
enactment-row lookups in `exception_occurs/3` and
`terminate_enactment/4`, returning structured
`{:error, {:fatal_persistence_failed | :terminate_persistence_failed, ctx}}`
on missing-row instead of crashing.
- Snapshot writes use `Repo.insert/2` (no bang).
- New `insert/2` and `get/2` entries in the `Repo` wrapper.
## Tests
- `Runner.ErrorsTest` covers tier classification, `error_code` lookups,
`to_persisted_reason` mapping for every Tier 2 module, and
`build_exception` round-trips.
- `WorkitemTransitionTest` exercises every branch of
`call_enactment/3` `:exit` translation: `:noproc` / `:timeout` /
`:shutdown` / `:normal` / `:killed` / `:calling_self` / catch-all,
plus the `whereis` pre-check.
- `EnactmentLogTest` rounds every reason in
`Runner.Exception.__reasons__/0` through `EnactmentLog.build_exception/3`
to catch enum drift.
- `EnactmentTest` adds a `lifecycle :start telemetry metadata` describe
block verifying fresh-boot vs post-snapshot `resumed` / `replayed_steps`.
## Misc
- Tighten `CatchingUp.apply/2` spec from `pos_integer()` to
`non_neg_integer()` (the empty-stream case is reachable).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>1 parent a738f4c commit 1378f7a
31 files changed
Lines changed: 2593 additions & 183 deletions
File tree
- lib/coloured_flow
- definition/colour_set
- expression
- runner
- enactment
- exceptions
- exception
- storage
- test/coloured_flow/runner
- enactment
- storage/schemas
Large diffs are not rendered by default.
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| |||
0 commit comments