feat(scripts): add witness caching for multi and cost_estimator #776
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Add witness caching to reduce 2-hour bottleneck in multi.rs and cost_estimator scripts.
Cache WitnessData (not SP1Stdin) to disk with keys (chain_id, start_block, end_block).
New features:
- Witness cache module with save/load functions using rkyv serialization
- cfg_if conditional compilation for EigenDA support
- Three cache flags: --cache (default), --use-cache (load only), --save-cache (force regenerate)
- Multi-script integration with cache status messages
- Cost estimator parallel cache loads per batch range
- Comprehensive documentation on cache usage and management
Cache location: data/{chain_id}/witness-cache/{start_block}-{end_block}.bin
DA compatibility: Ethereum/Celestia compatible, EigenDA separate cache
Files changed:
- utils/host/src/witness_cache.rs (NEW)
- utils/host/src/lib.rs, Cargo.toml
- scripts/utils/src/lib.rs
- scripts/prove/bin/multi.rs
- scripts/utils/bin/cost_estimator.rs
- book/advanced/cost-estimation-tools.md (renamed from cost-estimator.md)
- book/SUMMARY.md
fakedev9999
force-pushed
the
fakedev9999/cache-proving-input
branch
from
7d51eaa to
b2a2f7a
Compare
fakedev9999
changed the title
- Add SP1Stdin cache functions using bincode (DA-agnostic) - cost_estimator.rs now caches SP1Stdin instead of WitnessData - Remove WitnessDataType type constraint that caused CI failures - Fix race condition in multi.rs by using match pattern with graceful fallback SP1Stdin is the same type regardless of which DA witness generator produced it, so it works with generic host types. This fixes the CI type mismatch error when running with --features celestia/eigenda.
Switch both multi.rs and cost_estimator.rs to use SP1Stdin caching instead of WitnessData caching. This fixes CI failures when running with different DA feature flags (celestia, eigenda). SP1Stdin is DA-agnostic - it's the same type regardless of which witness generator produced it. This means cache files now work across all DA types (Ethereum, Celestia, EigenDA). Changes: - Update multi.rs to cache SP1Stdin using bincode - Simplify witness_cache.rs to only contain SP1Stdin functions - Remove eigenda feature flag from utils/host (no longer needed) - Update documentation to reflect SP1Stdin caching
Farhad-Shabani
requested changes
Jan 26, 2026
- Use tracing macros instead of println/eprintln in multi.rs - Fix DA compatibility docs: clarify that cache files are compatible between Ethereum ↔ Celestia, but NOT with EigenDA - Simplify cache flags: remove --use-cache and --save-cache, keep only --cache for simpler UX
Contributor
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-v3.x v3.x
# Navigate to the new working tree
cd .worktrees/backport-v3.x
# Create a new branch
git switch --create backport-776-to-v3.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7bfa64372cea807e7547e8afc8a9778be40c079e
# Push it to GitHub
git push --set-upstream origin backport-776-to-v3.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-v3.xThen, create a pull request where the |
fakedev9999
added a commit
that referenced
this pull request
Jan 30, 2026
* feat(witness-cache): add caching for witness generation
Add witness caching to reduce 2-hour bottleneck in multi.rs and cost_estimator scripts.
Cache WitnessData (not SP1Stdin) to disk with keys (chain_id, start_block, end_block).
New features:
- Witness cache module with save/load functions using rkyv serialization
- cfg_if conditional compilation for EigenDA support
- Three cache flags: --cache (default), --use-cache (load only), --save-cache (force regenerate)
- Multi-script integration with cache status messages
- Cost estimator parallel cache loads per batch range
- Comprehensive documentation on cache usage and management
Cache location: data/{chain_id}/witness-cache/{start_block}-{end_block}.bin
DA compatibility: Ethereum/Celestia compatible, EigenDA separate cache
Files changed:
- utils/host/src/witness_cache.rs (NEW)
- utils/host/src/lib.rs, Cargo.toml
- scripts/utils/src/lib.rs
- scripts/prove/bin/multi.rs
- scripts/utils/bin/cost_estimator.rs
- book/advanced/cost-estimation-tools.md (renamed from cost-estimator.md)
- book/SUMMARY.md
* fix(witness-cache): use SP1Stdin caching for cost_estimator to fix CI
- Add SP1Stdin cache functions using bincode (DA-agnostic)
- cost_estimator.rs now caches SP1Stdin instead of WitnessData
- Remove WitnessDataType type constraint that caused CI failures
- Fix race condition in multi.rs by using match pattern with graceful fallback
SP1Stdin is the same type regardless of which DA witness generator
produced it, so it works with generic host types. This fixes the CI
type mismatch error when running with --features celestia/eigenda.
* refactor(witness-cache): unify on SP1Stdin caching for DA-agnosticism
Switch both multi.rs and cost_estimator.rs to use SP1Stdin caching
instead of WitnessData caching. This fixes CI failures when running
with different DA feature flags (celestia, eigenda).
SP1Stdin is DA-agnostic - it's the same type regardless of which
witness generator produced it. This means cache files now work across
all DA types (Ethereum, Celestia, EigenDA).
Changes:
- Update multi.rs to cache SP1Stdin using bincode
- Simplify witness_cache.rs to only contain SP1Stdin functions
- Remove eigenda feature flag from utils/host (no longer needed)
- Update documentation to reflect SP1Stdin caching
* refactor(witness-cache): address PR review comments
- Use tracing macros instead of println/eprintln in multi.rs
- Fix DA compatibility docs: clarify that cache files are compatible
between Ethereum ↔ Celestia, but NOT with EigenDA
- Simplify cache flags: remove --use-cache and --save-cache,
keep only --cache for simpler UX
(cherry picked from commit 7bfa643)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds witness caching to skip the 2-hour witness generation bottleneck in
multi.rsandcost_estimatorscripts. Cache files are saved todata/{chain_id}/witness-cache/{start_block}-{end_block}.binusing rkyv serialization.Changes
--cache(recommended),--use-cache(load only),--save-cache(force regenerate)Testing
Run multi or cost_estimator with
--cacheflag to test caching behavior. Verify cache files are created at expected location.