docs: mainnet operational runbook (admin custody, launch sequence, incident escalation) - #208
docs: mainnet operational runbook (admin custody, launch sequence, incident escalation)#208Iker2522 wants to merge 2 commits into
Conversation
…escalation (drydocs#205) DEPLOYMENT.md already covers parameter selection (V1_MAINNET_PARAMETERS.md, drydocs#190) and the on-chain admin/resolver mechanics. RESOLVER_GOVERNANCE.md already covers resolver key custody, onboarding, and offboarding in depth. Neither covers the admin role's own key custody, a concrete go/no-go launch procedure, or incident escalation specifics -- this document is additive, covering exactly that gap, not a rewrite of either existing document. Explicitly framed throughout (and especially at the top) as prep work written ahead of the audit clearing, per SECURITY.md's hard blocker, the same way V1_MAINNET_PARAMETERS.md was written ahead of live testnet data: useful to have ready, not a signal mainnet deployment is authorized now. Also fixes a real, unrelated staleness bug found while researching this: DEPLOYMENT.md's 'Rotating the admin key' section still documented v1's old single-step set_admin, which no longer exists in this contract (replaced by the two-step propose_admin/accept_admin flow CONTRACT.md already documents correctly). Corrected it to match, since the new runbook's admin-custody section would otherwise contradict the existing admin-rotation docs it links to. Verified: read the actual propose_admin/accept_admin/set_paused/ set_stall_timeout/reclaim_stalled_dispute implementations in contracts/tholos/src/lib.rs directly (not just existing docs) to ground the custody and incident-escalation guidance in what the contract actually does, confirmed every internal doc cross-reference (10 anchored links across DEPLOYMENT.md, RESOLVER_GOVERNANCE.md, V1_MAINNET_PARAMETERS.md, and CONTRACT.md) resolves to a real heading id in the compiled mdbook output rather than assuming the anchor slugs, and confirmed mdbook build docs (v0.4.48, matching CI) succeeds. One link (to reclaim_stalled_dispute) points to the source doc comment instead of CONTRACT.md, since that function isn't documented there yet -- a separate, pre-existing gap out of this issue's scope.
collinsezedike
left a comment
There was a problem hiding this comment.
Good structure and correctly scoped against DEPLOYMENT.md/RESOLVER_GOVERNANCE.md. Two factual accuracy issues, both inline.
| shown there. | ||
| 8. **Verify on-chain state matches what was signed off on** before | ||
| announcing the contract id publicly or pointing any real value at it: | ||
| read back `bond_amount`, `challenge_window_secs`, `finalize_reward_bps`, |
There was a problem hiding this comment.
These read-only calls don't exist. contracts/tholos/src/lib.rs's only public read entrypoint is get_assertion_state(id), which returns an Assertion struct with no admin, challenge_window, finalize_reward_bps, or resolvers-at-rest fields (resolvers stays empty until a dispute happens). There's no get_admin, get_resolvers, get_bond_amount, or get_config. An operator following this step during an actual launch has no on-chain way to do the verification as written.
| [V1_MAINNET_PARAMETERS.md](V1_MAINNET_PARAMETERS.md) and | ||
| [BOND_SIZING.md](BOND_SIZING.md). These (except the resolver committee) | ||
| cannot be changed after `initialize` except `bond_amount` and | ||
| `finalize_reward_bps`'s own admin setters going forward — get them right |
There was a problem hiding this comment.
finalize_reward_bps has no admin setter. Grepping contracts/tholos/src/lib.rs shows only three admin setters, set_paused, set_bond_amount, set_stall_timeout, and finalize_reward_bps is written once, inside initialize. It's permanently fixed at deployment, same as challenge_window_secs and the resolver committee size. Please correct this so an operator doesn't treat getting it right pre-launch as less critical than it is.
collinsezedike caught two things, both verified directly against contracts/tholos/src/lib.rs before fixing: 1. Step 8 claimed bond_amount, challenge_window_secs, finalize_reward_bps, the resolver set, and the admin address could be read back via on-chain calls. They can't: get_assertion_state(id) is the contract's only public read entrypoint, and it returns per-assertion state, not configuration. initialize also emits no event. Rewrote the step to describe the only verification that actually exists: reading back the initialize transaction's own submitted arguments, not querying live state that has no getter. 2. Step 2 implied finalize_reward_bps has an admin setter alongside bond_amount. Grepping lib.rs confirms only three admin setters exist (set_paused, set_bond_amount, set_stall_timeout); finalize_reward_bps is written once inside initialize and permanently fixed, same as challenge_window_secs and the resolver committee size. Corrected. Also proactively grepped the rest of the document for any other getter/setter claims before considering this done, rather than fixing only the two flagged lines in isolation.
collinsezedike
left a comment
There was a problem hiding this comment.
Both prior findings are fixed, and the verification step rewrite is genuinely better than what was there before. One new issue introduced by that same rewrite, inline.
| 2. **Finalize and sign off on parameters** (`bond_amount`, | ||
| `challenge_window_secs`, `finalize_reward_bps`, resolver committee) per | ||
| [V1_MAINNET_PARAMETERS.md](V1_MAINNET_PARAMETERS.md) and | ||
| [BOND_SIZING.md](BOND_SIZING.md). Only `bond_amount` has an admin setter |
There was a problem hiding this comment.
This says only bond_amount has an admin setter while listing resolver committee in the same sentence, but the resolver committee is also changeable post-deployment via the admin's update_resolvers call, which Part 1 of this same document already lists as an admin lever. An operator reading this step in isolation could conclude the committee is permanently fixed like challenge_window_secs and finalize_reward_bps, when it isn't. Please carve resolver committee out of that sentence, or note update_resolvers alongside set_bond_amount.
Summary
DEPLOYMENT.md already covers parameter selection (V1_MAINNET_PARAMETERS.md, #190) and the on-chain admin/resolver mechanics. RESOLVER_GOVERNANCE.md already covers resolver key custody, onboarding, and offboarding in depth. Neither covers the admin role's own key custody, a concrete go/no-go launch procedure, or incident escalation specifics — this PR adds MAINNET_RUNBOOK.md, covering exactly that gap, not a rewrite of either existing document.
Explicitly framed throughout (and especially at the top) as prep work written ahead of the audit clearing, per SECURITY.md's hard blocker — useful to have ready, not a signal mainnet deployment is authorized now.
What changed
New docs/src/MAINNET_RUNBOOK.md: admin key custody (including a finding from reading the contract directly — propose_admin requires the current admin's signature, so there's no on-chain recovery from a genuinely lost/destroyed admin key), an 11-step go/no-go launch sequence, and an incident decision tree (set_paused vs. resolver rotation vs. reclaim_stalled_dispute vs. doing nothing).
Fixed a real, unrelated staleness bug found while researching this: DEPLOYMENT.md's "Rotating the admin key" section still documented the old single-step set_admin, which no longer exists — the contract now uses the two-step propose_admin/accept_admin flow CONTRACT.md already documents correctly. Corrected it so the new runbook doesn't contradict it.
Test plan
Read the actual propose_admin/accept_admin/set_paused/set_stall_timeout/reclaim_stalled_dispute implementations in contracts/tholos/src/lib.rs directly to ground the guidance in real behavior, not assumptions
Verified all 10 internal anchored links against the actual compiled mdbook HTML output (not assumed slug format) — one (reclaim_stalled_dispute) points to the source doc comment instead of CONTRACT.md, since that function isn't documented there yet (pre-existing gap, out of scope here)
mdbook build docs (v0.4.48, same as CI) builds cleanly
No contract or test changes — docs only
Closes #205