feat: optimize pruning performance and improve configuration#3
Open
dntjd1097 wants to merge 4 commits intoosmosis-labs:masterfrom
Open
feat: optimize pruning performance and improve configuration#3dntjd1097 wants to merge 4 commits intoosmosis-labs:masterfrom
dntjd1097 wants to merge 4 commits intoosmosis-labs:masterfrom
Conversation
- Use single target height instead of array for memory efficiency - Add --disable-fast-node flag (default: fast node enabled) - Fix logging by using actual logger instead of NopLogger - Implement parallel store pruning for better performance - Improve error logging with structured fields
- Simplify logic to directly pass disableFastNode to SetIAVLDisableFastNode - Remove confusing inverted if-else branches - Now correctly disables fast node when --disable-fast-node flag is used - Default behavior (no flag) enables fast node as intended
- Reorganize LevelDB options with categorized comments for better readability - Adjust block size from 48KB to 32KB for optimal I/O efficiency - Adjust compaction table size from 24MB to 16MB to reduce compaction frequency - Group options by purpose (speed optimizations, compression) - Add inline value comments for clarity (e.g., 64MB cache) - Maintain consistent optimization across both pruneAppState and pruneTMData functions Performance improvements: - 128MB write buffer (default 4MB) - reduces compaction frequency - 32KB block size (default 4KB) - improves I/O efficiency - 16MB compaction table (default 2MB) - reduces compaction overhead - 2000 open file cache (default 500) - reduces file open/close overhead - 64MB block cache (default 8MB) - improves read performance
This reverts commit 4e5fb38.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces several performance optimizations and improvements to the pruning functionality, focusing on memory efficiency, logging, and configuration flexibility. Performance testing shows significant improvements from 1-2 hours to ~5 minutes for a 50GB database.
Changes
1. Optimize Pruning Performance
Memory efficiency: Use single target height instead of array of heights
latestHeight - versionsin memoryImproved logging: Replace
NopLoggerwith actual logger2. Add IAVL Fast Node Configuration
New flag:
--disable-fast-node(default:false)Rationale: This is a pruning tool, not a database upgrade tool
Simplified logic: Direct pass-through of flag value to
SetIAVLDisableFastNode3. Enhanced Error Handling
base,targetfor block pruning)Performance Testing
Test Environment
Test Results
Key Improvements
Limitations
Testing
--disable-fast-nodeflag behaviors testedUsage Examples
Breaking Changes
None. All changes are backward compatible.
Related Issues
N/A