Skip to content

Commit 1378f7a

Browse files
fahchenclaude
andcommitted
feat(runner): error handling foundation and Tier 2 fatal funnel
Combined error-handling P0 implementation per `error_handling_design.md` (included). Squashes the foundation (PR #82) and storage funnel (PR #83) work into a single commit on top of `main` (which now carries the RuntimeCpnet, snapshot-debounce, hibernate_after, $callers, and async RepoCase changes from #76 / #77 / #79 / #80). ## Errors facade and exception vocabulary - New `ColouredFlow.Runner.Errors` central classifier: `tier/1`, `error_code/1`, `lifecycle?/1`, `to_persisted_reason/1`, `build_exception/2`. Documented as best-effort static dispatch — for context-sensitive classification subscribers should rely on the telemetry metadata emitted at the call site. - Stable `error_code` field on existing public exceptions (`NonLiveWorkitem`, `InvalidWorkitemTransition`, `UnboundActionOutput`, `UnsufficientTokensToConsume`, `ColourSetMismatch`, `Expression.InvalidResult`). - New caller-facing exceptions: `EnactmentNotRunning`, `EnactmentTimeout`, `EnactmentCallFailed`, `StoragePersistenceFailed`. - New Tier 2 enactment-fatal exceptions: `StateDrift`, `SnapshotCorrupt`, `ReplayFailed`, `EnactmentDataMissing`, `CpnetCorrupt`. - Extend `Runner.Exception.__reasons__/0` (`Ecto.Enum` for `enactment_logs.exception.reason`) with the five new fatal reasons. ## Caller-safe wrapper - `WorkitemTransition.call_enactment/3` translates the full `GenServer.call/3` `:exit` surface (`:noproc`, `:timeout`, `:shutdown`, `:normal`, `:nodedown`, `:killed`, `:calling_self`, catch-all) into typed exceptions returned via `{:error, exception}`. - `Enactment.Supervisor.terminate_enactment/2` and `Runner.terminate_enactment/2` reuse the same wrapper; return type widened from `:ok` to `:ok | {:error, Exception.t()}` (documented breaking change in `Runner` `@moduledoc`). - `Registry.whereis/1` for explicit pid lookup. ## Tier 2 fatal-stop funnel - New private `Enactment.to_exception/3`: persists the enactment as `:exception` via `Storage.exception_occurs/3`, emits a lifecycle `:exception` telemetry event with full metadata (`tier`, `lifecycle`, `severity`, `source_phase`, `exception_reason`, `error_code`, `exception`, `degraded`), and stops the GenServer with `{:shutdown, {:fatal, reason}}` so supervisor restart intensity is not consumed. - The existing termination-criteria fatal path is folded into the funnel. - New `perform_termination/4` for explicit / implicit / force termination success path; emits structured `{:shutdown, {:terminated, type}}`. - Caller storage failures from `start` / `complete` / `withdraw` / `produce` / `terminate` route through the funnel as `:state_drift` (synthesised `StateDrift` exception threaded back to the caller). - Bootstrap and async snapshot write failures log and continue (not fatal); the async snapshot path layers persistence-error handling on top of the existing mailbox-drain debounce semantics. - `terminate/2` recognises `{:shutdown, {:fatal, _}}`, `{:shutdown, {:terminated, _}}`, `{:fatal_persistence_failed, _}` and emits stable `:stop` event metadata (no more `"unknown"` placeholder). ## Cross-callback rescue - `populate_state` boot is split into `load_initial_snapshot/1`, `replay_occurrences/2`, `maybe_persist_bootstrap_snapshot/2` so each rescue scope attributes errors precisely: snapshot decode -> `:snapshot_corrupt`, occurrence replay -> `:replay_failed`, missing enactment row -> `:enactment_data_missing`. - Both `:calibrate_workitems` `handle_continue` clauses now rescue flow-decode value-shape errors -> `:cpnet_corrupt` and `Ecto.NoResultsError` -> `:enactment_data_missing`. - `handle_call({:complete_workitems, _}, _, _)` likewise rescues missing / corrupt flow errors at the callback boundary, replying with the matching typed exception before the GenServer stops. - `catchup_snapshot/2` now returns `{snapshot, replayed_steps}`; populate_state emits `resumed: true | false` and `replayed_steps` metadata on the `:start` lifecycle event so subscribers can distinguish a fresh boot from a crash recovery without breaking existing `[:start]` listeners. ## Storage contract - Storage callbacks that previously raised on `Multi` rollback or unexpected row counts now return labelled error tuples (`{:error, {reason, ctx}}`). Affected: `exception_occurs/3`, `insert_enactment/1`, `terminate_enactment/4`, `produce_workitems/2`, `start_workitems/2`, `withdraw_workitems/2`, `complete_workitems/4`, `take_enactment_snapshot/2`. - `Storage.Default` switched to `Ecto.Multi.run` + `Repo.get/2` for the enactment-row lookups in `exception_occurs/3` and `terminate_enactment/4`, returning structured `{:error, {:fatal_persistence_failed | :terminate_persistence_failed, ctx}}` on missing-row instead of crashing. - Snapshot writes use `Repo.insert/2` (no bang). - New `insert/2` and `get/2` entries in the `Repo` wrapper. ## Tests - `Runner.ErrorsTest` covers tier classification, `error_code` lookups, `to_persisted_reason` mapping for every Tier 2 module, and `build_exception` round-trips. - `WorkitemTransitionTest` exercises every branch of `call_enactment/3` `:exit` translation: `:noproc` / `:timeout` / `:shutdown` / `:normal` / `:killed` / `:calling_self` / catch-all, plus the `whereis` pre-check. - `EnactmentLogTest` rounds every reason in `Runner.Exception.__reasons__/0` through `EnactmentLog.build_exception/3` to catch enum drift. - `EnactmentTest` adds a `lifecycle :start telemetry metadata` describe block verifying fresh-boot vs post-snapshot `resumed` / `replayed_steps`. ## Misc - Tighten `CatchingUp.apply/2` spec from `pos_integer()` to `non_neg_integer()` (the empty-stream case is reachable). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
1 parent a738f4c commit 1378f7a

31 files changed

Lines changed: 2593 additions & 183 deletions

error_handling_design.md

Lines changed: 688 additions & 0 deletions
Large diffs are not rendered by default.

lib/coloured_flow/definition/colour_set/colour_set_mismatch.ex

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ defmodule ColouredFlow.Definition.ColourSet.ColourSetMismatch do
88
typed_structor definer: :defexception, enforce: true do
99
field :colour_set, ColouredFlow.Definition.ColourSet.t()
1010
field :value, term()
11+
field :error_code, atom(), default: :colour_set_mismatch
1112
end
1213

1314
@impl Exception

lib/coloured_flow/expression/invalid_result.ex

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ defmodule ColouredFlow.Expression.InvalidResult do
1212
field :expression, Expression.t()
1313

1414
field :message, String.t()
15+
field :error_code, atom(), default: :expression_eval_failed
1516
end
1617

1718
@impl Exception

lib/coloured_flow/runner/enactment/catching_up.ex

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ defmodule ColouredFlow.Runner.Enactment.CatchingUp do
1818
@spec apply(
1919
current_markings :: [Marking.t()],
2020
occurrences :: Enumerable.t(Occurrence.t())
21-
) :: {steps :: pos_integer(), [Marking.t()]}
21+
) :: {steps :: non_neg_integer(), [Marking.t()]}
2222
def apply(current_markings, occurrences) do
2323
current_markings = to_map(current_markings)
2424

0 commit comments

Comments
 (0)