Skip to content

Handoff immutability policies, Phase 1: the policy mechanism#27

Merged
joelhelbling merged 3 commits into
mainfrom
feat/handoff-policy-mechanism
Jul 10, 2026
Merged

Handoff immutability policies, Phase 1: the policy mechanism#27
joelhelbling merged 3 commits into
mainfrom
feat/handoff-policy-mechanism

Conversation

@joelhelbling

Copy link
Copy Markdown
Owner

Part 1 of 4 for the 0.6.0 handoff-immutability release. Implements Phase 1 of
the implementation plan
(committed on this branch), which was produced from
the design doc via han planning.

What this does

Every value crossing a worker boundary is now governed by a handoff policy, applied at the consuming worker's intake:

policy task receives mutation in task
:frozen (new default) deeply frozen shared reference raises Shifty::PolicyViolation at the offending worker
:isolated private mutable deep copy (Marshal) works, stays local
:shared the raw reference (old behavior) works, leaks — the escape hatch

Declared per worker (policy: kwarg — the worker's contract), per pipeline (.with_policy(...) on a chain or Gang, policy: kwarg on Gang — default for undeclared workers), or globally (Shifty.configure { |c| c.default_policy = ... }). Precedence: worker > pipeline > global.

Design notes (differ from the design doc — empirically driven)

  • :isolated uses Marshal, not make_shareable(copy: true): the latter returns a frozen copy, which can't satisfy the mutable-scratch-copy contract.
  • IO is rejected proactively under :frozen (Shifty::UnshareableValue): Ractor.make_shareable does not reject IO — it silently freezes the live handle process-wide (spike verified; it froze $stdout mid-experiment).
  • Enforcement is at intake (one seam + a policy-governed supply proxy for mid-task supply.shift), so filter/batch/trailing workers and multi-yield splitter/source workers are covered with no task-body changes.

Also in this PR

  • trailing_worker now hands off a snapshot instead of its live closure array (the exact aliasing bug class the policy exists to catch).
  • side_worker mode: :hardened deprecated → policy: :isolated (warning; removal at 1.0.0). Under :isolated, a side worker's mutations evaporate — the behavior the docs always wished for.
  • Errors consolidated under Shifty::Error; PolicyViolation carries worker/policy/receiver/value/cause with a receiver-reachability heuristic.
  • required_ruby_version >= 3.2; ostruct declared (not a default gem since Ruby 4); CI matrix 2.6/2.7/3.0 → 3.2/3.3/3.4; codeclimate-test-reporter dropped (pinned simplecov to 2016).
  • Three existing specs that asserted mutation-leaking behavior migrated per plan; they double as migration worked examples.

Verification

  • 130 examples, 0 failures (was 91 at baseline); standardrb clean; gem build succeeds.
  • Nothing releases from this PR — version stays 0.5.0 until the Phase 4 release PR.

Phases 2–4 (testing harness & mutation detector; benchmarks & #freeze!; docs/wiki/release) follow as separate PRs.

Produced by han plan-implementation: specialist round (concurrency,
architecture, testing, junior-dev stress test), decision log (22 decisions),
iteration history, and the phased implementation plan for 0.6.0.
Every value crossing a worker boundary is now governed by a handoff
policy, applied at the consuming worker's intake:

- :frozen (new default): deep-freeze via Ractor.make_shareable, zero-copy;
  mutation anywhere raises Shifty::PolicyViolation at the offending worker,
  with a receiver heuristic (the value itself / reachable from it /
  possibly unrelated) and the original FrozenError as cause.
- :isolated: private mutable deep copy via Marshal. (Empirical spike:
  make_shareable(copy: true) returns a *frozen* copy, so it cannot satisfy
  the mutable-scratch contract the spec assumed.)
- :shared: the raw reference, unchanged — the escape hatch for IO handles
  and intentional shared mutation.

Declaration surface: policy: kwarg on Worker (the worker's contract),
with_policy(...) on a composed chain or Gang plus policy: kwarg on Gang
(pipeline default for undeclared workers), and
Shifty.configure { |c| c.default_policy = ... } (global). Precedence:
worker > pipeline > global.

IO values are rejected proactively under :frozen with
Shifty::UnshareableValue — make_shareable does NOT reject IO; it silently
freezes the live handle process-wide (verified by spike).

Also: policy-governed supply proxy covers mid-task supply.shift calls
(filter/batch/trailing); trailing_worker hands off a snapshot instead of
its live closure array; side_worker mode: :hardened is deprecated, mapped
to policy: :isolated (removal at 1.0.0); errors consolidated under
Shifty::Error; worker name: kwarg for diagnostics; required_ruby_version
>= 3.2; ostruct declared (no longer a default gem in Ruby 4); CI matrix
now 3.2/3.3/3.4; codeclimate-test-reporter dropped.
Eleven findings from the six-agent han review were confirmed by an
adversarial validation pass (with live reproductions); all confirmed
code findings are fixed here:

- Validate policy names eagerly at every declaration site (Worker.new,
  Gang.new, Shifty.configure) via Policy.validate!, matching with_policy;
  a typo now fails where it was written, not at first shift inside a
  Fiber.
- Remove @resolved_policy memoization so what effective_policy reports
  is always what intake enforces, even after late policy changes.
- Rescue PolicyError in Worker#shift and discard the terminated Fiber,
  so a caller that rescues a PolicyViolation can keep shifting; the
  violating value is consumed and the pipeline continues with the next.
- Govern Gang#shift's criteria-bypass path through the entry worker's
  intake, matching Worker#shift's bypass behavior.
- Persist a Gang's declared pipeline policy and apply it to workers
  appended after the declaration.
- Nil-guard Gang#supply so with_policy on an empty gang works.
- Cycle-guard the with_policy supply-chain walk (self-referential pipes
  previously hung).
- Deprecation-warn on ALL side_worker mode: values, with a message that
  names what the user actually typed; mode: :hardened maps to
  policy: :isolated without double-warning.
- Delegate side_worker's scratch copy to Policy::Isolated (removes the
  duplicated Marshal round-trip).
- Bound the PolicyViolation reachability walk (iterative, 50k-node cap)
  so a deeply nested value can't SystemStackError during error
  construction and mask the violation.
- New specs: eager validation, violation recovery, gang bypass/append/
  precedence/empty-gang, self-pipe termination, mode: :normal warning,
  Struct branch of the receiver heuristic.
@joelhelbling

Copy link
Copy Markdown
Owner Author

Han code review — Phase 1 (medium change, 6 agents + adversarial validation)

Automated checks: 141 examples / 0 failures · standardrb clean · gem build succeeds.

Agents: junior-developer, adversarial-security-analyst, test-engineer, edge-case-explorer, structural-analyst, concurrency-analyst; consolidated findings re-attacked by an adversarial-validator with live reproductions.

Security

No proven vulnerabilities. Marshal round-trips confirmed in-process only (never external bytes); dependencies current.

Findings and outcomes (all fixed in 2d72412 unless noted)

ID Sev Finding Outcome
WARN-001 Warning Policy names validated eagerly by with_policy but lazily by Worker.new/Gang.new/configure FixedPolicy.validate! at every declaration site
WARN-003 Warning @resolved_policy memoization could diverge from what effective_policy reports Fixed — memoization removed
WARN-004 Warning Gang#append after a policy declaration silently dropped the policy Fixed — gang persists its declared policy
WARN-005 Warning PolicyViolation killed the Fiber; rescue-and-reshift raised FiberError Fixedshift discards the dead fiber; pipeline continues with the next value
WARN-006 Warning Gang#shift criteria-bypass skipped policy (unlike Worker's bypass) Fixed — bypass routes through the entry worker's intake
WARN-007 Warning with_policy on an empty Gang raised NoMethodError Fixed — nil-guarded Gang#supply (validator corrected the repro: needs a valid policy name)
WARN-008 Warning mode: only half-deprecated; non-:hardened values silently dropped Fixed — all mode: uses warn; message names what was typed (also resolves SUGG-003)
WARN-009 Warning Task-facing supply arg is now Policy::Supply (only #shift) — breaking contract change Documented — goes in the Phase 4 CHANGELOG/migration guide
SUGG-002 Suggestion Self-referential pipe hung with_policy's walk Fixed — cycle guard
SUGG-001 Suggestion Unbounded recursion in the receiver-reachability walk could SystemStackError on deep values Fixed — iterative walk, 50k-node cap
SUGG-004 Suggestion Untested: Struct heuristic branch; gang-vs-member precedence Fixed — specs added
WARN-002→SUGG Suggestion side_worker duplicated the Marshal round-trip (validator refuted the claimed error-surface regression — intake wraps first) Fixed — delegates to Policy::Isolated

YAGNI (advisory, intentionally not corrected)

Reachability graph-walk machinery (serves the loud/local-violations goal; test-backed); Configuration class ceremony (justified by the Phase 2/3 roadmap).

What's good (per agents)

The intake seam correctly governs all four supply.shift sites with zero task-body changes; trailing_worker's snapshot fix lands the exact aliasing bug class the spec predicted; errors.rb consolidation removed pre-existing duplication.

Recommendation: merge. No critical findings; all confirmed warnings fixed and regression-tested.

@joelhelbling joelhelbling merged commit 2bd4dec into main Jul 10, 2026
3 checks passed
@joelhelbling joelhelbling deleted the feat/handoff-policy-mechanism branch July 10, 2026 22:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant