Release chaos suite: keep cutover MVP harness; expected-fail vs crash

[lionakhnazarov]

Summary

PR #4109 cutover MVP built a network testing harness (private E1-TC, 12-op exact-image fleet, real 100/90/51). Overall MVP-FAIL. Several fails are product/ops, not “inconvenient tests”: clock on ETH loss, seating vs signing threshold, SIGTERM drain, PRIOR rollback onto R1 storage.

Proposal: keep this as a per-release chaos suite, and use the same shape later for FROST/ROAST (similar security risk, larger).

Pass rule

Outcome	Verdict
Happy path works (DKG, heartbeat, challenge)	pass
Injected fault → documented timeout / refuse / clock_unavailable / clean exit	pass (predictable failure)
Injected fault → panic, hang, gate still “open” with no chain, silent ignore of SIGTERM	fail

Example: run DKG on the new release, then randomly kill N of 100 seats. If the remaining set is below signing threshold and the ceremony times out cleanly, that is success. If nodes crash on an unhandled error, that is a product bug.

Findings from this rehearsal (file separately)

• Participation clock stays available when Ethereum subscription is dead #4245 — ETH disconnect does not fail participation clock (S10)
• 89-seat R1 cohort still completes heartbeat TSS (90 vs 51 boundary) #4246 — 89 R1 seats still complete TSS; 90 vs 51 (S07)
• SIGTERM drain does not enter quiescing or exit within grace #4247 — SIGTERM drain does not quiesce/exit in grace (S12)
• PRIOR cannot heartbeat-sign R1-created 90-seat wallet storage (rollback) #4248 — PRIOR cannot heartbeat-sign R1 90-seat storage (S13)

Do not greenwash these by loosening needles. They are the interesting cases.

Scope sketch

1. Happy-path scenarios on every keep-core candidate (DKG, heartbeat, challenge).
2. Chaos mode: partition ETH, SIGTERM, kill N operators, remount volumes / rollback binary.
3. Same harness later for FROST/ROAST mixed-binary transition.

Harness today lives outside keep-core (scripts/cutover-mvp/ in the lab tree). This issue is the product-side tracking item so release QA has a home.

References

• Findings write-up: 20260817-pr4109-cutover-mvp-findings.md
• Freeze: overall MVP-FAIL; S18 freeze-only passed; cannot issue plan GO

[linear]

ENG-744