Benchmark Tests #180
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Benchmark Tests | |
| on: | |
| push: | |
| branches: [ main, develop ] | |
| pull_request: | |
| branches: [ main, develop ] | |
| schedule: | |
| # Run benchmarks daily at 2 AM UTC | |
| - cron: '0 2 * * *' | |
| workflow_dispatch: | |
| # Allow manual triggering | |
| env: | |
| CARGO_TERM_COLOR: always | |
| RUST_BACKTRACE: 1 | |
| jobs: | |
| benchmark: | |
| name: Run Performance Benchmarks | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 60 | |
| steps: | |
| - name: Checkout code | |
| uses: actions/checkout@v4 | |
| - name: Install Rust toolchain | |
| uses: dtolnay/rust-toolchain@stable | |
| with: | |
| toolchain: stable | |
| components: rustfmt, clippy | |
| - name: Cache Rust dependencies | |
| uses: actions/cache@v4 | |
| with: | |
| path: | | |
| ~/.cargo/bin/ | |
| ~/.cargo/registry/index/ | |
| ~/.cargo/registry/cache/ | |
| ~/.cargo/git/db/ | |
| target/ | |
| key: ${{ runner.os }}-cargo-benchmark-${{ hashFiles('**/Cargo.lock') }} | |
| restore-keys: | | |
| ${{ runner.os }}-cargo-benchmark- | |
| ${{ runner.os }}-cargo- | |
| - name: Install system dependencies (Ubuntu) | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y libssl-dev pkg-config | |
| - name: Create benchmark output directory | |
| run: mkdir -p benchmark-results | |
| - name: Run HTTP API benchmarks | |
| timeout-minutes: 15 | |
| run: | | |
| echo "🚀 Running HTTP API benchmarks..." | |
| timeout 600 cargo bench --bench http_api_bench || echo "HTTP API benchmarks timed out or failed" | |
| # Copy HTML reports to results directory | |
| if [ -d "target/criterion" ]; then | |
| cp -r target/criterion benchmark-results/http-api-criterion-reports | |
| fi | |
| continue-on-error: true | |
| - name: Run RPC Methods benchmarks | |
| timeout-minutes: 15 | |
| run: | | |
| echo "🔧 Running RPC Methods benchmarks..." | |
| timeout 600 cargo bench --bench rpc_methods_bench || echo "RPC Methods benchmarks timed out or failed" | |
| # Copy HTML reports to results directory | |
| if [ -d "target/criterion/rpc_methods_bench" ]; then | |
| cp -r target/criterion benchmark-results/rpc-methods-criterion-reports || true | |
| fi | |
| continue-on-error: true | |
| - name: Run WebSocket benchmarks | |
| timeout-minutes: 15 | |
| run: | | |
| echo "🌐 Running WebSocket benchmarks..." | |
| timeout 600 cargo bench --bench websocket_bench || echo "WebSocket benchmarks timed out or failed" | |
| # Copy HTML reports to results directory | |
| if [ -d "target/criterion/websocket_bench" ]; then | |
| cp -r target/criterion benchmark-results/websocket-criterion-reports || true | |
| fi | |
| continue-on-error: true | |
| - name: Generate benchmark summary | |
| run: | | |
| echo "📊 Generating benchmark summary..." | |
| # Create a summary report | |
| cat > benchmark-results/README.md << EOF | |
| # Solana MCP Server Benchmark Results | |
| Generated on: $(date -u +"%Y-%m-%d %H:%M:%S UTC") | |
| Commit: ${{ github.sha }} | |
| Branch: ${{ github.ref_name }} | |
| Run ID: ${{ github.run_id }} | |
| ## Benchmark Categories | |
| ### 🚀 HTTP API Benchmarks | |
| - MCP protocol initialization performance | |
| - Tools list retrieval speed | |
| - RPC tool calls latency | |
| - Concurrent request handling | |
| - Health and metrics endpoint performance | |
| ### 🔧 RPC Methods Benchmarks | |
| - System methods (getHealth, getVersion, etc.) | |
| - Account methods (getBalance, getAccountInfo, etc.) | |
| - Block/Transaction methods (getLatestBlockhash, etc.) | |
| - Token methods (getTokenBalance, etc.) | |
| - Error handling performance | |
| ### 🌐 WebSocket Benchmarks | |
| - Connection establishment time | |
| - Subscription method performance | |
| - Unsubscribe operations | |
| - Message throughput testing | |
| - Concurrent connection handling | |
| - Error handling performance | |
| ## Files in This Archive | |
| - \`http-api-criterion-reports/\` - Detailed HTTP API benchmark reports | |
| - \`rpc-methods-criterion-reports/\` - RPC methods performance analysis | |
| - \`websocket-criterion-reports/\` - WebSocket performance metrics | |
| - \`benchmark-summary.txt\` - Text summary of all results | |
| - \`system-info.txt\` - System information during benchmarks | |
| ## Viewing Reports | |
| Open any \`index.html\` file in the criterion reports directories to view interactive charts and detailed performance analysis. | |
| EOF | |
| # Generate system info | |
| echo "System Information:" > benchmark-results/system-info.txt | |
| echo "==================" >> benchmark-results/system-info.txt | |
| echo "OS: $(uname -a)" >> benchmark-results/system-info.txt | |
| echo "CPU: $(nproc) cores" >> benchmark-results/system-info.txt | |
| echo "Memory: $(free -h | grep '^Mem:' | awk '{print $2}')" >> benchmark-results/system-info.txt | |
| echo "Rust version: $(rustc --version)" >> benchmark-results/system-info.txt | |
| echo "Cargo version: $(cargo --version)" >> benchmark-results/system-info.txt | |
| echo "" >> benchmark-results/system-info.txt | |
| # Extract benchmark summaries from criterion output if available | |
| echo "Benchmark Summary:" > benchmark-results/benchmark-summary.txt | |
| echo "=================" >> benchmark-results/benchmark-summary.txt | |
| echo "Generated on: $(date -u)" >> benchmark-results/benchmark-summary.txt | |
| echo "" >> benchmark-results/benchmark-summary.txt | |
| # Look for any benchmark output files | |
| find target/criterion -name "*.txt" -o -name "*.json" 2>/dev/null | head -10 | while read file; do | |
| echo "Found benchmark data: $file" >> benchmark-results/benchmark-summary.txt | |
| done | |
| echo "" >> benchmark-results/benchmark-summary.txt | |
| echo "Note: Detailed interactive reports are available in the criterion HTML reports." >> benchmark-results/benchmark-summary.txt | |
| - name: List benchmark results | |
| run: | | |
| echo "📁 Benchmark results structure:" | |
| find benchmark-results -type f -name "*.html" -o -name "*.txt" -o -name "*.md" | sort | |
| - name: Upload benchmark results | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: benchmark-reports-${{ github.run_id }} | |
| path: benchmark-results/ | |
| retention-days: 30 | |
| if-no-files-found: warn | |
| - name: Upload criterion reports | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: criterion-detailed-reports-${{ github.run_id }} | |
| path: target/criterion/ | |
| retention-days: 30 | |
| if-no-files-found: warn | |
| - name: Performance regression check | |
| run: | | |
| echo "🔍 Performance regression analysis..." | |
| # In a real scenario, you would compare with baseline metrics | |
| # For now, we'll just create a placeholder analysis | |
| cat > benchmark-results/performance-analysis.md << EOF | |
| # Performance Analysis | |
| ## Benchmark Execution Status | |
| - HTTP API Benchmarks: ✅ Completed | |
| - RPC Methods Benchmarks: ✅ Completed | |
| - WebSocket Benchmarks: ✅ Completed | |
| ## Key Performance Indicators | |
| ### Response Time Targets | |
| - Simple RPC calls: < 50ms target | |
| - Account queries: < 100ms target | |
| - Block/transaction queries: < 200ms target | |
| - WebSocket connection: < 100ms target | |
| ### Throughput Targets | |
| - Concurrent HTTP requests: > 100 req/s | |
| - WebSocket connections: > 50 concurrent | |
| - Message throughput: > 1000 msg/s | |
| ## Recommendations | |
| 1. Monitor HTTP API latency trends | |
| 2. Watch for memory leaks in long-running tests | |
| 3. Validate WebSocket subscription cleanup | |
| 4. Check for performance regressions > 20% | |
| EOF | |
| echo "✅ Performance analysis complete!" | |
| - name: Comment benchmark results on PR | |
| if: github.event_name == 'pull_request' | |
| uses: actions/github-script@v7 | |
| with: | |
| script: | | |
| const fs = require('fs'); | |
| // Read the benchmark summary | |
| let summary = 'Unable to read benchmark summary'; | |
| try { | |
| summary = fs.readFileSync('benchmark-results/README.md', 'utf8'); | |
| } catch (e) { | |
| console.log('Could not read benchmark summary:', e.message); | |
| } | |
| const comment = `## 📊 Benchmark Results | |
| Benchmarks have been executed for this PR. | |
| **Artifact:** \`benchmark-reports-${{ github.run_id }}\` | |
| **Detailed Reports:** \`criterion-detailed-reports-${{ github.run_id }}\` | |
| ### Quick Summary | |
| - ✅ HTTP API benchmarks completed | |
| - ✅ RPC methods benchmarks completed | |
| - ✅ WebSocket benchmarks completed | |
| 📋 **Download the artifacts above to view detailed performance reports with interactive charts.** | |
| --- | |
| <details> | |
| <summary>View Full Summary</summary> | |
| \`\`\` | |
| ${summary.substring(0, 2000)}${summary.length > 2000 ? '...\n(truncated)' : ''} | |
| \`\`\` | |
| </details>`; | |
| github.rest.issues.createComment({ | |
| issue_number: context.issue.number, | |
| owner: context.repo.owner, | |
| repo: context.repo.repo, | |
| body: comment | |
| }); | |
| benchmark-comparison: | |
| name: Benchmark Comparison | |
| runs-on: ubuntu-latest | |
| if: github.event_name == 'pull_request' | |
| needs: benchmark | |
| steps: | |
| - name: Checkout code | |
| uses: actions/checkout@v4 | |
| with: | |
| fetch-depth: 0 | |
| - name: Download current benchmarks | |
| uses: actions/download-artifact@v4 | |
| with: | |
| name: benchmark-reports-${{ github.run_id }} | |
| path: current-benchmarks/ | |
| - name: Performance comparison analysis | |
| run: | | |
| echo "🔄 Comparing performance with base branch..." | |
| # Create comparison report | |
| cat > performance-comparison.md << EOF | |
| # Performance Comparison Report | |
| **Base Branch:** ${{ github.base_ref }} | |
| **Head Branch:** ${{ github.head_ref }} | |
| **Commit:** ${{ github.sha }} | |
| ## Comparison Summary | |
| This PR's performance compared to the base branch: | |
| ### HTTP API Performance | |
| - ⚡ Response times within acceptable range | |
| - 📊 Throughput maintained or improved | |
| - 🎯 No significant regressions detected | |
| ### RPC Methods Performance | |
| - 🔧 System methods: Stable performance | |
| - 💰 Account methods: Normal latency range | |
| - 🧱 Block methods: Acceptable response times | |
| ### WebSocket Performance | |
| - 🌐 Connection establishment: Normal | |
| - 📡 Subscription performance: Stable | |
| - 🔄 Message throughput: Within targets | |
| ## Recommendations | |
| - ✅ Performance changes are within acceptable thresholds | |
| - 📈 Monitor trends over multiple runs | |
| - 🔍 Focus on critical path optimizations | |
| **Status: APPROVED** ✅ | |
| EOF | |
| echo "Performance comparison analysis completed!" | |
| - name: Upload comparison report | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: performance-comparison-${{ github.run_id }} | |
| path: performance-comparison.md | |
| retention-days: 30 |