A professional multi blockchain node performance benchmarking framework with comprehensive QPS testing, real-time monitoring, intelligent bottleneck detection, and advanced visualization reporting.
- Multi-Mode QPS Testing: Quick (15+min), Standard (90+min), and Intensive (8+hr) testing modes
- Cross-Platform Compatibility: Supports 8 mainstream blockchain nodes (Solana, Ethereum, BSC, Base, Polygon, Scroll, Starknet, Sui) on AWS, other clouds, IDC, or local Linux environments
- Real Transaction Data Testing: Fetches active accounts from your blockchain node and generates test targets using single or mixed RPC methods
- Multi-Layered Performance Monitoring: Professional monitoring system with 4 specialized data streams
- Unified metrics (79 fields): CPU, Memory, EBS, Network, ENA, Block Height, QPS
- Monitoring overhead tracking (20 fields): Self-monitoring and impact analysis
- ENA deep monitoring (15 fields): AWS-specific network performance analysis
- Blockchain health tracking (7 fields): Node sync status and data loss detection
- Dual Bottleneck Monitoring Mechanism:
- Real-time Detection: 8-dimensional monitoring with 5-scenario judgment logic to avoid false positives
- Resource bottleneck + Node healthy β False positive (reset counter)
- RPC performance violation (success rate < 95% OR latency > 1000ms) β True bottleneck (necessary condition)
- Any bottleneck + Node unhealthy β True bottleneck (3 consecutive times)
- Node persistently unhealthy alone β Node failure (stop immediately)
- All normal β Continue testing
- Offline Analysis: Multi-dimensional deep analysis after test completion
- Time window analysis (Β±30 seconds around bottleneck) for focused root cause investigation
- Performance cliff analysis to identify QPS degradation patterns
- EBS performance deep dive with AWS baseline comparison
- CPU-EBS correlation analysis for resource bottleneck identification
- RPC method performance profiling and optimization recommendations
- Real-time Detection: 8-dimensional monitoring with 5-scenario judgment logic to avoid false positives
- AWS Deep Integration: EBS performance baselines, ENA network monitoring
- Professional Visualization: 32 professional charts and comprehensive HTML reports
Before running the framework, you must configure the following parameters:
# 1. RPC Endpoint (Required)
LOCAL_RPC_URL="http://localhost:8899" # Your blockchain node RPC endpoint
# 2. Blockchain Type (Required)
BLOCKCHAIN_NODE="Solana" # Supported: Solana, Ethereum, BSC, Base, Polygon, Scroll, Starknet, Sui
# 3. Blockchain Process Names (Required for monitoring)
BLOCKCHAIN_PROCESS_NAMES=(
"agave-validator" # Your actual blockchain node process name
"solana-validator" # Add all possible process names
"validator"
)
# Check your process name with: ps aux | grep -i validator# 4. DATA Device Configuration (Required)
LEDGER_DEVICE="nvme1n1" # DATA device name (check with 'lsblk')
DATA_VOL_TYPE="io2" # Options: "gp3" | "io2" | "instance-store"
DATA_VOL_MAX_IOPS="30000" # Your EBS volume's provisioned IOPS
DATA_VOL_MAX_THROUGHPUT="700" # Your EBS volume's throughput (MiB/s)
# 5. ACCOUNTS Device (Optional, but recommended for complete monitoring)
ACCOUNTS_DEVICE="nvme2n1" # ACCOUNTS device name
ACCOUNTS_VOL_TYPE="io2" # Options: "gp3" | "io2" | "instance-store"
ACCOUNTS_VOL_MAX_IOPS="30000" # ACCOUNTS volume's provisioned IOPS
ACCOUNTS_VOL_MAX_THROUGHPUT="700" # ACCOUNTS volume's throughput (MiB/s)
# 6. Network Configuration (Required for AWS environments)
NETWORK_MAX_BANDWIDTH_GBPS=25 # Your instance's network bandwidth (Gbps)Quick Configuration Check:
# Verify your blockchain process name
ps aux | grep -i validator
ps aux | grep -i agave
# Verify your EBS devices
lsblk
# Check your EBS volume configuration in AWS Console:
# EC2 β Volumes β Select your volume β Details tab
# - IOPS: Provisioned IOPS value
# - Throughput: Provisioned throughput value
# Check your instance network bandwidth:
# EC2 β Instance Types β Search your instance type β NetworkingConfiguration File Locations:
config/config_loader.sh- RPC endpoint, blockchain type, process namesconfig/user_config.sh- EBS devices, network bandwidth, monitoring intervals
Note: If you don't configure these parameters correctly, the framework will use default values which may not match your actual hardware, leading to inaccurate performance analysis.
The fastest way is to use the bundled dependency installer:
# Interactive install (asks before each step that needs sudo)
bash scripts/install_deps.sh
# Non-interactive (CI / Docker / unattended VMs)
bash scripts/install_deps.sh --yes
# Audit only β check what's missing without installing anything
bash scripts/install_deps.sh --check
# Show all flags
bash scripts/install_deps.sh --helpThe installer covers:
- System packages (
sysstat bc jq net-tools procps) β auto-detects apt/yum/dnf/apk - Python packages (from
requirements.txt) β handles PEP 668 (Debian 12+) automatically - vegeta v12.12.0 β pinned version, SHA256 verified, installed to
~/bin/vegeta
Supported distros: Ubuntu, Debian, RHEL, CentOS, Rocky, AlmaLinux, Amazon Linux, Fedora, Alpine.
Manual install (if you prefer step-by-step control)
# Install Vegeta v12.12.0 (QPS testing tool)
wget https://github.com/tsenart/vegeta/releases/download/v12.12.0/vegeta_12.12.0_linux_amd64.tar.gz
tar -xzf vegeta_12.12.0_linux_amd64.tar.gz
sudo mv vegeta /usr/local/bin/
vegeta -version # Verify installation
# Install system monitoring tools (framework hard deps β all required)
sudo apt-get install -y sysstat bc jq net-tools procps
# sysstat β iostat/mpstat/sar | bc β arithmetic in shell scripts
# jq β JSON parsing | net-tools β netstat | procps β ps/top
# Install Python and virtual environment support
sudo apt-get install python3 python3-venv
# Create virtual environment
python3 -m venv node-env
# Activate virtual environment
source node-env/bin/activate
# Check Python version (requires Python 3.8+)
python3 --version
# Install Python dependencies (from project root directory)
pip3 install -r requirements.txt
# Note: on Debian 12+ (PEP 668), if you skip the venv above, use:
# pip3 install --user --break-system-packages -r requirements.txt
# Verify all tools are installed
which vegeta # QPS testing tool
which iostat # I/O monitoring tool
which mpstat # CPU monitoring tool
which sar # Network monitoring tool# Quick test (15+ minutes) - can run directly
./blockchain_node_benchmark.sh --quick
# Standard test (90+ minutes) - recommended to use screen
screen -S benchmark_$(date +%m%d_%H%M)
./blockchain_node_benchmark.sh --standard
# β οΈ MUST press Ctrl+a then d to detach before closing SSH!
# Intensive test (up to 8 hours) - MUST use screen/tmux
screen -S benchmark_$(date +%m%d_%H%M)
./blockchain_node_benchmark.sh --intensive
# β οΈ MUST press Ctrl+a then d to detach before closing SSH!- Use
screenortmux - Detach the session (Ctrl+a, then d for screen) before closing SSH
- Otherwise the test will stop when SSH disconnects!
See Best Practices for detailed instructions.
# Custom intensive test with specific parameters
./blockchain_node_benchmark.sh --intensive \
--initial-qps 1000 \
--max-qps 10000 \
--step-qps 500 \
--duration 300 \
--mixed # Use mixed RPC method testingblockchain-node-benchmark/
βββ π― Core Execution Layer
β βββ blockchain_node_benchmark.sh # Main entry script
β βββ master_qps_executor.sh # QPS testing engine
β βββ common_functions.sh # Shared function library
βββ βοΈ Configuration Management
β βββ config_loader.sh # Configuration loader
β βββ system_config.sh # System configuration
βββ π Monitoring Data Layer
β βββ unified_monitor.sh # Unified monitor
β βββ bottleneck_detector.sh # Bottleneck detector
β βββ monitoring_coordinator.sh # Monitoring coordinator
βββ π¬ Analysis Processing Layer
β βββ comprehensive_analysis.py # Comprehensive analyzer
β βββ qps_analyzer.py # QPS analyzer
β βββ rpc_deep_analyzer.py # RPC deep analyzer
βββ π Visualization Layer
β βββ report_generator.py # HTML report generator
β βββ performance_visualizer.py # Performance visualization engine
β βββ advanced_chart_generator.py # Advanced chart generator
βββ π οΈ Tools & Utilities
βββ benchmark_archiver.sh # Test result archiver
βββ ebs_bottleneck_detector.sh # EBS bottleneck detector
βββ target_generator.sh # Test target generator
Comprehensive documentation is available in the docs/ directory:
- 4-layer modular architecture design
- Component interaction and data flow
- 32 professional charts breakdown
- System integration points
- Complete data file structure and field definitions
- 79-field performance data CSV format
- 20-field monitoring overhead CSV format
- Data flow architecture and file naming conventions
- JSON format specifications for test results
- 4-layer configuration system (User/System/Internal/Dynamic)
- EBS volume configuration (gp3/io2/instance-store)
- Network and ENA settings
- Blockchain-specific parameters
- Dual-layer monitoring architecture
- 79 performance metrics collection (updated)
- Self-monitoring and overhead analysis
- AWS standard conversion formulas
- Single vs Mixed RPC testing modes
- Multi-blockchain support (Solana/Ethereum/BSC/Base/Polygon/Scroll/Starknet/Sui)
- RPC method configuration
- Real transaction data testing
RPC and Blockchain Settings (config/config_loader.sh):
LOCAL_RPC_URL="http://localhost:8899"
BLOCKCHAIN_NODE="Solana"
BLOCKCHAIN_PROCESS_NAMES=(
"agave-validator"
"solana-validator"
"validator"
)EBS Device Configuration (config/user_config.sh):
# DATA device (required)
LEDGER_DEVICE="nvme1n1"
DATA_VOL_TYPE="io2" # io2/gp3/instance-store
DATA_VOL_MAX_IOPS="30000"
DATA_VOL_MAX_THROUGHPUT="700" # MiB/s
# ACCOUNTS device (optional)
ACCOUNTS_DEVICE="nvme2n1"
ACCOUNTS_VOL_TYPE="io2" # io2/gp3/instance-store
ACCOUNTS_VOL_MAX_IOPS="30000"
ACCOUNTS_VOL_MAX_THROUGHPUT="700" # MiB/s
# Network configuration
NETWORK_MAX_BANDWIDTH_GBPS=25 # GbpsNote: ACCOUNTS device is optional. If not configured, the framework will only monitor the DATA device.
Bottleneck Detection Thresholds (config/internal_config.sh):
BOTTLENECK_CPU_THRESHOLD=85
BOTTLENECK_MEMORY_THRESHOLD=90
BOTTLENECK_EBS_UTIL_THRESHOLD=90
BOTTLENECK_EBS_LATENCY_THRESHOLD=50
NETWORK_UTILIZATION_THRESHOLD=80Monitoring Intervals (config/user_config.sh):
MONITOR_INTERVAL=5 # Default monitoring interval (seconds)
HIGH_FREQ_INTERVAL=1 # High-frequency monitoring interval
ULTRA_HIGH_FREQ_INTERVAL=0.5 # Ultra-high-frequency monitoring interval| Mode | Duration | QPS Range | Step Size | Use Case |
|---|---|---|---|---|
| Quick | 15+ minutes | 1000-3000 | 500 QPS | Basic performance verification |
| Standard | 90+ minutes | 20000-50000 | 500 QPS | Comprehensive performance evaluation |
| Intensive | Up to 8 hours | 50000-unlimited | 250 QPS | Intelligent bottleneck detection |
- Timestamp: Unified timestamp (1 field)
- CPU: Usage, I/O wait, system calls (6 fields)
- Memory: Usage, available memory, cache (3 fields)
- EBS Storage: IOPS, throughput, latency, utilization (21 fields per device, 42 fields for 2 devices)
- Network: Bandwidth utilization, PPS, connections (10 fields)
- ENA Network: Allowance exceeded, bandwidth limits (6 fields, AWS only)
- Monitoring Overhead: System impact metrics (2 fields)
- Block Height: Local vs mainnet sync status (6 fields)
- QPS Performance: Current QPS, latency, availability (3 fields)
Total: 79 fields (with ENA on AWS) or 73 fields (without ENA on non-AWS)
The framework generates multiple specialized CSV files for fine-grained analysis:
1. Performance Metrics (performance_YYYYMMDD_HHMMSS.csv - 79 fields)
- Unified performance data combining all system metrics
- Real-time collection at configurable intervals (default 5s)
- Used for comprehensive performance analysis and correlation studies
2. Monitoring Overhead (monitoring_overhead_YYYYMMDD_HHMMSS.csv - 20 fields)
- Monitoring system resource consumption (CPU, memory, process count)
- Blockchain node resource consumption
- System-level resource statistics (cores, memory, disk, cache, buffers)
- Used for monitoring impact analysis and overhead optimization
3. ENA Network Details (ena_network_YYYYMMDD_HHMMSS.csv - 15 fields, AWS only)
- Network interface statistics (rx/tx bytes, packets)
- ENA allowance tracking (bandwidth in/out, PPS, connection tracking, link-local)
- Network limitation status (network_limited, pps_limited, bandwidth_limited)
- Used for AWS ENA-specific bottleneck detection and network performance analysis
4. Block Height Monitor (block_height_monitor_YYYYMMDD_HHMMSS.csv - 7 fields)
- Local and mainnet block height tracking
- Block height difference and sync status
- Node health indicators (local_health, mainnet_health)
- Data loss detection flag
- Used for blockchain node sync analysis and health monitoring
- CPU Bottleneck: Threshold 85%
- Memory Bottleneck: Threshold 90%
- EBS Bottleneck: IOPS/Throughput/Utilization > 90% baseline
- Network Bottleneck: Bandwidth/PPS utilization > 80%
- ENA Bottleneck: Allowance limits exceeded
- RPC Success Rate: < 95% (Necessary Condition)
- RPC Latency: P99 > 1000ms (Necessary Condition)
- RPC Error Rate: > 5%
View complete sample reports generated from real test data (Standard mode, 90+ minutes):
- HTML Report - Interactive HTML with all charts
- PDF Report - Printable PDF version
Advanced Analysis Charts (9 charts):
pearson_correlation_analysis.png- Pearson Correlation Analysislinear_regression_analysis.png- Linear Regression Analysisnegative_correlation_analysis.png- Negative Correlation Analysisena_limitation_trends.png- ENA Limitation Trendsena_connection_capacity.png- ENA Connection Capacityena_comprehensive_status.png- ENA Comprehensive Statuscomprehensive_correlation_matrix.png- Comprehensive Correlation Matrixperformance_trend_analysis.png- Performance Trend Analysisperformance_correlation_heatmap.png- Performance Correlation Heatmap
EBS Professional Charts (7 charts):
ebs_aws_capacity_planning.png- AWS Capacity Planning Analysisebs_iostat_performance.png- Iostat Performance Analysisebs_bottleneck_correlation.png- Bottleneck Correlation Analysisebs_performance_overview.png- EBS Performance Overviewebs_bottleneck_analysis.png- EBS Bottleneck Analysisebs_aws_standard_comparison.png- EBS AWS Standard Comparisonebs_time_series_analysis.png- EBS Time Series Analysis
Core Performance Charts (11 charts):
performance_overview.png- Performance Overviewcpu_ebs_correlation_visualization.png- CPU-EBS Correlation Analysisdevice_performance_comparison.png- Device Performance Comparisonawait_threshold_analysis.png- I/O Latency Threshold Analysismonitoring_overhead_analysis.png- Monitoring Overhead Analysisqps_trend_analysis.png- QPS Trend Analysisresource_efficiency_analysis.png- Resource Efficiency Analysisbottleneck_identification.png- Bottleneck Identificationblock_height_sync_chart.png- Block Height Sync Chartsmoothed_trend_analysis.png- Smoothed Trend Analysisutil_threshold_analysis.png- Utilization Threshold Analysis
Additional Analysis Charts (5 charts):
resource_distribution_chart.png- Resource Distribution Chartmonitoring_impact_chart.png- Monitoring Impact Analysiscomprehensive_analysis_charts.png- Comprehensive Analysis Chartsperformance_cliff_analysis.png- Performance Cliff Analysisqps_performance_analysis.png- QPS Performance Analysis
- System-Level Bottleneck Analysis: Bottleneck detection results and optimization recommendations
- Performance Summary: Test overview and key performance metrics
- Configuration Status Check: System configuration validation
- Blockchain Node Sync Analysis: Block height monitoring and sync status
- EBS Performance Analysis Results: Storage performance deep dive with AWS baseline comparison
- Performance Chart Gallery: All 32 professional visualization charts organized by category
- Monitoring Overhead Analysis: Comprehensive monitoring system impact analysis
- CPU-EBS Correlation Analysis: Resource correlation and bottleneck identification
# Run standard test
./blockchain_node_benchmark.sh --standard
# View results
ls reports/
# comprehensive_analysis_report.html
# performance_overview.png
# cpu_ebs_correlation_visualization.png
# ... (other chart files)# Custom intensive test with specific parameters
./blockchain_node_benchmark.sh --intensive \
--initial-qps 2000 \
--max-qps 15000 \
--step-qps 1000 \
--mixed # Use mixed RPC method testing# Check QPS testing engine status
./core/master_qps_executor.sh --status
# Check monitoring system status
./monitoring/monitoring_coordinator.sh status
# View test history
./tools/benchmark_archiver.sh --list# Recommended: Install specific version v12.12.0
wget https://github.com/tsenart/vegeta/releases/download/v12.12.0/vegeta_12.12.0_linux_amd64.tar.gz
tar -xzf vegeta_12.12.0_linux_amd64.tar.gz
sudo mv vegeta /usr/local/bin/
vegeta -version # Should show: Version: 12.12.0
# Alternative: Install via package manager (may be older version)
# Ubuntu/Debian
sudo apt-get install vegeta# Check detected network interface
echo $NETWORK_INTERFACE
# List all network interfaces
ip link show
# Manually specify network interface if auto-detection fails
export NETWORK_INTERFACE="eth0" # Replace with your interface name
# Or add to config/user_config.sh:
# NETWORK_INTERFACE="eth0"
# Common interface names:
# - AWS: eth0, eth1
# - Other clouds: eth0, ens3, ens5
# - Local: eth0, enp0s3, wlan0# Install sysstat package
sudo apt-get install sysstat # Ubuntu/Debian
sudo yum install sysstat # CentOS/RHEL# Reinstall dependencies
pip3 install --upgrade -r requirements.txt
# Check specific packages
python3 -c "import matplotlib, pandas, numpy; print('All packages OK')"# Grant execution permissions
chmod +x blockchain_node_benchmark.sh
chmod +x core/master_qps_executor.sh
chmod +x monitoring/monitoring_coordinator.shAll logs are stored in blockchain-node-benchmark-result/current/logs/:
- QPS Test Log:
master_qps_executor.log- QPS test progress and results - Monitoring Log:
unified_monitor.log- System monitoring data - Bottleneck Detection:
bottleneck_detector.log- Bottleneck detection events - EBS Analysis:
ebs_bottleneck_detector.log- EBS performance analysis - Performance Data:
performance_YYYYMMDD_HHMMSS.csv- Raw performance metrics (79 fields) - Monitoring Overhead:
monitoring_overhead_YYYYMMDD_HHMMSS.csv- Monitoring system overhead (20 fields) - ENA Network:
ena_network_YYYYMMDD_HHMMSS.csv- ENA network detailed metrics (15 fields, AWS only) - Block Height Monitor:
block_height_monitor_YYYYMMDD_HHMMSS.csv- Block height sync tracking (7 fields)
If your terminal disconnects during testing, you can reconnect and view progress:
# View QPS test progress in real-time
tail -f blockchain-node-benchmark-result/current/logs/master_qps_executor.log
# Check current test status
ps aux | grep vegeta | grep -v grep
# View latest performance data
tail -20 blockchain-node-benchmark-result/current/logs/performance_latest.csv
# Check if bottleneck detected
cat /dev/shm/blockchain-node-benchmark/bottleneck_status.json | jq '.'
# View completed test results
ls -lt blockchain-node-benchmark-result/current/vegeta_results/ | head -10Once testing is complete, all results are archived:
# View archived results
ls -lt blockchain-node-benchmark-result/archives/
# Access specific test run
cd blockchain-node-benchmark-result/archives/run_XXX_YYYYMMDD_HHMMSS/
# View logs
cat logs/master_qps_executor.log
# View reports
open reports/performance_report_en_*.html# Step 1: Create screen session with unique name
screen -S benchmark_$(date +%m%d_%H%M)
# Example: benchmark_1030_2200
# Step 2: Start test
./blockchain_node_benchmark.sh --intensive
# Step 3: β οΈ IMPORTANT - Detach before closing SSH
# Press: Ctrl+a, then press d
# You should see: [detached from xxx.benchmark_1030_2200]
# Step 4: Now you can safely close SSH
exit
# Step 5: Reconnect anytime
# List all screen sessions
screen -ls
# Reconnect to specific session
# If shows "(Detached)" - use simple reconnect:
screen -r benchmark_1030_2200
# Or use PID:
screen -r 12345
# If shows "(Attached)" - use force reconnect:
screen -d -r 12345
# This detaches any existing connection and reconnects youCommon Issues:
# Problem: Multiple sessions with same name
screen -ls
# Shows: 13813.benchmark, 19327.benchmark, 54872.benchmark
# Solution 1: Use PID to reconnect to latest
screen -r 13813
# Solution 2: Clean up old sessions
screen -X -S 19327 quit
screen -X -S 54872 quit
# Solution 3: Kill all and start fresh
killall screenWhy detach is critical: If you close SSH without detaching, the test will stop!
# Step 1: Create tmux session
tmux new -s benchmark
# Step 2: Start test
./blockchain_node_benchmark.sh --intensive
# Step 3: β οΈ IMPORTANT - Detach before closing SSH
# Press: Ctrl+b, then press d
# Step 4: Reconnect anytime
tmux attach -t benchmarknohup ./blockchain_node_benchmark.sh --intensive > test.log 2>&1 &
# View progress: tail -f test.log
# List historical tests
./tools/benchmark_archiver.sh --list
# Compare test results
./tools/benchmark_archiver.sh --compare run_001 run_002
# Clean up old tests
./tools/benchmark_archiver.sh --cleanup --days 30# Use Python analysis components
from analysis.comprehensive_analysis import ComprehensiveAnalyzer
analyzer = ComprehensiveAnalyzer("reports")
analyzer.run_comprehensive_analysis("logs/performance_latest.csv")# Run multiple test modes
for mode in quick standard intensive; do
echo "Running $mode test..."
./blockchain_node_benchmark.sh --$mode
sleep 60 # Wait for system recovery
doneWe welcome contributions! By contributing to this project, you agree to the following:
By submitting a pull request, you agree that:
- Your contributions will be licensed under both AGPL 3.0 and our Commercial License
- You grant the project maintainers the right to use your contributions in commercial versions
- You have the right to submit the contributions (no third-party IP conflicts)
# Clone the repository
git clone <repository-url>
cd blockchain-node-benchmark
# Install development dependencies
pip3 install -r requirements.txt
# Verify installation
python3 --version
bash --version- Fork and Branch: Create a feature branch from
main - Code Style: Follow PEP 8 for Python, use shellcheck for bash scripts
- Documentation: Update relevant documentation
- Commit Messages: Use clear, descriptive commit messages
- Pull Request: Submit PR with detailed description
- Add data collection logic in
monitoring/unified_monitor.sh - Update
generate_csv_header()function to add new fields - Add corresponding analysis logic in Python analysis scripts
- Update visualization components to generate related charts
Open an issue with the question label for any contribution-related questions.
This project is dual-licensed:
- Free for personal, academic, and open-source projects
- Modifications must be open-sourced
- Network use requires source disclosure
- See LICENSE for full terms
- Required for commercial/proprietary use
- Closed-source integration allowed
- No AGPL obligations
- Enterprise support available
- See LICENSE.COMMERCIAL for details
Contact: Open a GitHub Issue with label commercial-license for commercial licensing inquiries