backport: feat(scripts): add witness caching for multi and cost_estimator #791
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat(witness-cache): add caching for witness generation
Add witness caching to reduce 2-hour bottleneck in multi.rs and cost_estimator scripts.
Cache WitnessData (not SP1Stdin) to disk with keys (chain_id, start_block, end_block).
New features:
- Witness cache module with save/load functions using rkyv serialization
- cfg_if conditional compilation for EigenDA support
- Three cache flags: --cache (default), --use-cache (load only), --save-cache (force regenerate)
- Multi-script integration with cache status messages
- Cost estimator parallel cache loads per batch range
- Comprehensive documentation on cache usage and management
Cache location: data/{chain_id}/witness-cache/{start_block}-{end_block}.bin
DA compatibility: Ethereum/Celestia compatible, EigenDA separate cache
Files changed:
- utils/host/src/witness_cache.rs (NEW)
- utils/host/src/lib.rs, Cargo.toml
- scripts/utils/src/lib.rs
- scripts/prove/bin/multi.rs
- scripts/utils/bin/cost_estimator.rs
- book/advanced/cost-estimation-tools.md (renamed from cost-estimator.md)
- book/SUMMARY.md
* fix(witness-cache): use SP1Stdin caching for cost_estimator to fix CI
- Add SP1Stdin cache functions using bincode (DA-agnostic)
- cost_estimator.rs now caches SP1Stdin instead of WitnessData
- Remove WitnessDataType type constraint that caused CI failures
- Fix race condition in multi.rs by using match pattern with graceful fallback
SP1Stdin is the same type regardless of which DA witness generator
produced it, so it works with generic host types. This fixes the CI
type mismatch error when running with --features celestia/eigenda.
* refactor(witness-cache): unify on SP1Stdin caching for DA-agnosticism
Switch both multi.rs and cost_estimator.rs to use SP1Stdin caching
instead of WitnessData caching. This fixes CI failures when running
with different DA feature flags (celestia, eigenda).
SP1Stdin is DA-agnostic - it's the same type regardless of which
witness generator produced it. This means cache files now work across
all DA types (Ethereum, Celestia, EigenDA).
Changes:
- Update multi.rs to cache SP1Stdin using bincode
- Simplify witness_cache.rs to only contain SP1Stdin functions
- Remove eigenda feature flag from utils/host (no longer needed)
- Update documentation to reflect SP1Stdin caching
* refactor(witness-cache): address PR review comments
- Use tracing macros instead of println/eprintln in multi.rs
- Fix DA compatibility docs: clarify that cache files are compatible
between Ethereum ↔ Celestia, but NOT with EigenDA
- Simplify cache flags: remove --use-cache and --save-cache,
keep only --cache for simpler UX
(cherry picked from commit 7bfa643)
fakedev9999
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Backport of #776 to
release/v3.x.This adds SP1Stdin caching to skip the 2-hour witness generation bottleneck in
multi.rsandcost_estimatorscripts.Key Changes:
witness_cachemodule with unified SP1Stdin caching (bincode serialization)--cacheflag to enable cachingdata/{chain_id}/witness-cache/{start_block}-{end_block}-stdin.binOriginal PR
#776
(cherry picked from commit 7bfa643)