fix(settlement): reuse the prepared proof when retrying a failed submit - #639
Merged
Conversation
The settle loop re-ran the whole proving pipeline on every retry of a failed attempt, even when proving had succeeded and only the update_state submission failed. On the SP1 prover network every retry is a paid proving round: the mainnet enclave burned ~0.4 PROVE per minute (~24/hour) re-proving the identical range while its settlement account was short on STRK for the submission fee (2026-07-17 incident). Hold the proving result in the worker as a PreparedBatch keyed by the settle range: retries of the unchanged range skip straight to submission, a successful submission spends the payload, and a range change (the on-chain cursor moved) drops it before proving fresh. Reuse is sound because the payload is validity-stable for a fixed historical range — on-chain validation checks the proof against the range-derived commitment, not any timestamp embedded at proving time. The cache is deliberately in-memory: a restart clears it, which is also the escape hatch if a held proof is ever invalidated externally (e.g. a TEE-registry trust-root rotation). PiltoverClient::update_state now takes the input by reference so the worker keeps ownership across a failed submit (the generated PiltoverInput bindings don't derive Clone). The new proof_reuse tests drive prepare_batch/settle_batch with a counting mock backend, including the incident shape: submission fails against an unreachable Piltover endpoint after proving succeeded, and the retry must not prove again. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codec benchmark diff vs
|
| Benchmark | Baseline (ns) | Current (ns) | Δ |
|---|---|---|---|
CompiledClass(fixture)/compress |
2698471 | 2637056 | -2.28% |
CompiledClass(fixture)/decompress |
2819218 | 2779860 | -1.40% |
ExecutionCheckpoint/compress |
30 | 33 | +10.00% |
ExecutionCheckpoint/decompress |
23 | 25 | +8.70% |
PruningCheckpoint/compress |
30 | 33 | +10.00% |
PruningCheckpoint/decompress |
23 | 25 | +8.70% |
VersionedHeader/compress |
654 | 653 | -0.15% |
VersionedHeader/decompress |
835 | 890 | +6.59% |
StoredBlockBodyIndices/compress |
74 | 79 | +6.76% |
StoredBlockBodyIndices/decompress |
34 | 38 | +11.76% |
StorageEntry/compress |
144 | 149 | +3.47% |
StorageEntry/decompress |
141 | 157 | +11.35% |
ContractNonceChange/compress |
146 | 149 | +2.05% |
ContractNonceChange/decompress |
235 | 260 | +10.64% |
ContractClassChange/compress |
189 | 192 | +1.59% |
ContractClassChange/decompress |
254 | 281 | +10.63% |
ContractStorageEntry/compress |
163 | 157 | -3.68% |
ContractStorageEntry/decompress |
337 | 350 | +3.86% |
GenericContractInfo/compress |
134 | 135 | +0.75% |
GenericContractInfo/decompress |
104 | 110 | +5.77% |
Felt/compress |
82 | 73 | -10.98% |
Felt/decompress |
56 | 61 | +8.93% |
BlockHash/compress |
82 | 72 | -12.20% |
BlockHash/decompress |
56 | 60 | +7.14% |
TxHash/compress |
81 | 71 | -12.35% |
TxHash/decompress |
56 | 61 | +8.93% |
ClassHash/compress |
82 | 73 | -10.98% |
ClassHash/decompress |
56 | 60 | +7.14% |
CompiledClassHash/compress |
83 | 71 | -14.46% |
CompiledClassHash/decompress |
56 | 61 | +8.93% |
BlockNumber/compress |
46 | 33 | -28.26% |
BlockNumber/decompress |
24 | 24 | +0.00% |
TxNumber/compress |
46 | 33 | -28.26% |
TxNumber/decompress |
24 | 25 | +4.17% |
FinalityStatus/compress |
1 | 0 | -100.00% |
FinalityStatus/decompress |
10 | 10 | +0.00% |
TypedTransactionExecutionInfo/compress |
18030 | 14848 | -17.65% |
TypedTransactionExecutionInfo/decompress |
3709 | 3717 | +0.22% |
VersionedContractClass/compress |
380 | 364 | -4.21% |
VersionedContractClass/decompress |
802 | 832 | +3.74% |
MigratedCompiledClassHash/compress |
152 | 154 | +1.32% |
MigratedCompiledClassHash/decompress |
138 | 159 | +15.22% |
ContractInfoChangeList/compress |
1406 | 1408 | +0.14% |
ContractInfoChangeList/decompress |
2171 | 2278 | +4.93% |
BlockChangeList/compress |
614 | 605 | -1.47% |
BlockChangeList/decompress |
846 | 922 | +8.98% |
ReceiptEnvelope/compress |
31533 | 26585 | -15.69% |
ReceiptEnvelope/decompress |
6165 | 6504 | +5.50% |
TrieDatabaseValue/compress |
169 | 171 | +1.18% |
TrieDatabaseValue/decompress |
211 | 263 | +24.64% |
TrieHistoryEntry/compress |
287 | 335 | +16.72% |
TrieHistoryEntry/decompress |
245 | 307 | +25.31% |
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #639 +/- ##
==========================================
- Coverage 73.32% 68.19% -5.14%
==========================================
Files 209 335 +126
Lines 23132 47004 +23872
==========================================
+ Hits 16961 32053 +15092
- Misses 6171 14951 +8780 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ter restart A crash or restart between proving and settling discarded the prepared payload, forcing a fresh (paid) SP1 proving round for the identical range. The proof itself still exists on the prover network though — so persist just its network reference and recover it instead. - The moment proving yields a ProofId, record it as a PendingBatchProof (range + id) in a new singleton PendingSettlementProofs table — before update_state submission. Cleared when the batch settles. - On an in-memory miss, prepare_batch resolves a matching persisted reference via the new ProvingBackend::recover, which fetches the fulfilled proof from the network (SDK-side: the new Program::recover_proof — sp1_sdk wait_proof by request id, same plumbing as gen_raw_proof minus the paid request; SDK rev bump) and rebuilds the payload from a freshly built attestation. Only range-derived fields enter the payload, so a rebuilt attestation + recovered proof is byte-equivalent in everything on-chain validation checks. Mock proving reports recovery as unsupported (it is free anyway); any recovery failure falls back to fresh proving. - Escape hatch for externally invalidated proofs: an update_state that REVERTS in execution (e.g. a TEE-registry trust-root rotation between proving and submission) drops the payload and its persisted reference, so the retry proves fresh. Pre-execution failures (fees, nonce, transport) keep both — that's the incident case where reuse is the whole point. Table addition is version-neutral (same as SettlementProofs in #631). New proof_reuse tests cover restart recovery, unsupported/failed recovery fallback, stale-range references, and reference lifetime across failed submissions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the PreparedBatch cache field with structure: settle_batch now proves at most once and then loops on the update_state submission itself, holding the payload as a local scoped to the attempt. Every invalidation rule the cache needed becomes plain control flow — reuse while the range is unchanged is staying in the loop, spend-on-success is returning, drop-on-cursor-move is bailing out for the run loop to recompute, and drop-on-execution-revert is the one explicit early return. The persisted PendingBatchProof reference remains solely for crash recovery. The submission loop owns its own backoff, failure accounting, and the cursor-idempotency re-read, and stays shutdown-responsive by selecting on the (Unpin) oneshot receiver borrowed from the run loop, returning SettleOutcome::ShuttingDown. The outer loop's error branch now only handles terminal attempt failures (prove failure, execution revert, cursor moved). Log-line change for operators: transient submission failures now emit "Failed to submit state update; will retry with the same proof." instead of re-entering "Failed to settle block range" each cycle (that message now marks terminal attempt failures only). Docs updated. Tests now drive settle_batch's real submission-retry loop under a paused tokio clock (tokio test-util) against an unreachable Piltover endpoint, shutting it down after a bounded virtual time: one prove across many submission retries, restart recovery, recovery fallbacks, stale-reference skip, and reference lifetime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Option<ProofId> in recover's return was structurally dead: recovery happens BY a proof id, so a recovered payload can never lack one — the inner Option only existed because the signature mirrored prove's shape, where None legitimately means off-network proving. And returning the id at all was redundant: the caller reads it out of the persisted PendingBatchProof before calling recover. The backend now returns just the payload and settle_batch pairs it with the reference it resolved. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When
update_statesubmission fails after proving succeeded (fees, nonce, transient RPC), the settle loop retried the whole pipeline — including a fresh, paid SP1 proving round — every ~60s for the identical block range. Production incident (mainnet enclave, 2026-07-17): the settlement account was short on STRK for the submission fee, and the node burned ~0.4 PROVE per retry (~24 PROVE/hour) re-proving range 96–105 in a loop that could never succeed. A node restart had the same cost: the in-flight proof died with the process even though the prover network still held it.What
Prove once, retry the submission (
Worker::settle_batch): the payload is obtained at most once per attempt and lives as a local scoped to it; an inner loop retries only theupdate_statesubmission (own backoff, failure metrics, cursor-idempotency re-read, shutdown-responsive via the run loop's oneshot receiver). Reuse-while-unchanged, spend-on-success, drop-on-cursor-move, and drop-on-execution-revert are all control flow rather than cache state. Terminal conditions (prove failure, execution revert, on-chain cursor moved) returnErrto the run loop, which backs off and recomputes the range.PiltoverClient::update_statetakes&PiltoverInput(the generated bindings don't deriveClone).Recovery across restarts (persisted network reference): the moment proving yields a
ProofId, it is persisted as aPendingBatchProof(range + id) in a new singletonPendingSettlementProofstable — before submission — and cleared when the batch settles. On startup of an attempt, a matching reference is resolved through the newProvingBackend::recover, which fetches the fulfilled proof from the prover network (SDK-side: newProgram::recover_proof—sp1_sdkwait_proofby request id, the same plumbing asgen_raw_proofminus the paid request; SDK rev bump to cartridge-gg/amd-sev-snp-attestation-sdk@2685854) and rebuilds the payload from a freshly built attestation. Only range-derived fields enter the payload, so a rebuilt attestation + recovered proof is equivalent in everything on-chain validation checks. Mock proving reports recovery as unsupported (it's free anyway); recovery failure (expired/unknown id) falls back to fresh proving.Escape hatch: an
update_statethat reverts in execution (e.g. a TEE-registry trust-root rotation invalidated the proof between proving and submission) drops the payload and its persisted reference, so the next attempt proves fresh — retry-with-same-proof only applies to pre-execution failures (fees, nonce, transport), the incident class where it's wanted.Reuse is sound because the payload is validity-stable for a fixed historical range —
report_datais a pure function of the settled range, and Piltover validates the proof against the range-derived commitment, not any timestamp embedded at proving time. ThePendingSettlementProofstable addition is version-neutral (same asSettlementProofsin #631).proof_generation_secondsnow records only actual proving rounds.Operator-visible log change: transient submission failures emit
Failed to submit state update; will retry with the same proof.;Failed to settle block range; will retry.now marks terminal attempt failures only. Docs updated accordingly.Tests
The
proof_reusesuite drivessettle_batch's real submission-retry loop under a paused tokio clock against an unreachable Piltover endpoint (the incident shape), shut down after bounded virtual time: one prove across many submission retries with the persisted reference surviving, restart recovery from the reference, unsupported/failed recovery fallback to fresh proving, stale-range reference skip, prove-failure early return, and reference clearing.🤖 Generated with Claude Code