Skip to content

PERFORMANCE_BENCHMARKS_IMPLEMENTATION.md

Latest commit

 

History

History
262 lines (206 loc) · 7.6 KB

PERFORMANCE_BENCHMARKS_IMPLEMENTATION.md

File metadata and controls

262 lines (206 loc) · 7.6 KB
Raw

Performance Benchmarking Implementation Summary

What Was Added

A comprehensive performance benchmarking system for the Soroban Smart Block Explorer backend that automatically detects regressions in critical code paths with a 10% threshold in CI/CD.

Files Created

Core Framework (5 files)

  1. src/benchmarks/types.ts (47 lines)

    • TypeScript interfaces for benchmark data structures
    • BenchmarkResult, BenchmarkComparison, BenchmarkSuite types

  2. src/benchmarks/runner.ts (135 lines)

    • BenchmarkRunner class for high-resolution timing
    • Supports both sync and async benchmarking
    • Automatic warmup, GC collection, timeout protection
    • Statistics calculation (mean, median, stddev, percentiles)

  3. src/benchmarks/comparison.ts (206 lines)

    • BenchmarkStore class for persistent result storage
    • Regression detection with configurable thresholds
    • Markdown report generation
    • JSON export for CI/CD integration

  4. src/benchmarks/cli.ts (172 lines)

    • Command-line tool for benchmark management
    • Commands: compare, report, export, list, clean
    • Supports multi-suite analysis and reporting

Benchmark Suites (3 files, 12 benchmarks total)

  1. src/benchmarks/event-decoding.bench.ts (173 lines)

    • SEP-41 transfer event decoding (1ms target)
    • Complex generic event decoding (1ms target)
    • XDR parsing overhead (0.2ms target)

  2. src/benchmarks/api-responses.bench.ts (201 lines)

    • Transactions list endpoint (<100ms target)
    • Events list endpoint (<100ms target)
    • Contract detail endpoint (<150ms target)
    • Pagination overhead (<200ms target)

  3. src/benchmarks/rpc-calls.bench.ts (248 lines)

    • getEvents RPC overhead (<5ms target)
    • getTransaction RPC call (<3ms target)
    • getLedger RPC call (<3ms target)
    • Batch event processing (<20ms target)
    • Retry logic overhead (<5ms target)

CI/CD Integration (1 file)

  1. .github/workflows/benchmarks.yml (165 lines)
    • Runs on every push/PR to main and develop
    • Matrix testing (Node 18.x and 20.x)
    • Automatic regression detection
    • PR comments with results
    • Artifact uploads (30-day retention)
    • Main branch report generation

Documentation (1 file)

  1. BENCHMARKS.md (398 lines)
    • Complete usage guide
    • Architecture overview
    • Metric definitions
    • CLI reference
    • Best practices
    • Performance targets
    • Troubleshooting guide

Configuration Changes

package.json (7 new npm scripts)

"bench": "vitest run src/benchmarks/",
"bench:watch": "vitest watch src/benchmarks/",
"bench:event-decoding": "vitest run src/benchmarks/event-decoding.bench.ts",
"bench:api-responses": "vitest run src/benchmarks/api-responses.bench.ts",
"bench:rpc-calls": "vitest run src/benchmarks/rpc-calls.bench.ts",
"bench:compare": "ts-node src/benchmarks/cli.ts compare",
"bench:report": "ts-node src/benchmarks/cli.ts report"

Key Features

1. High-Resolution Performance Measurement

  • Uses Node.js perf_hooks API for nanosecond-level accuracy
  • Automatic JIT warmup and GC collection
  • Timeout protection and safety limits

2. Comprehensive Metrics

Each benchmark produces:

  • Mean, median, standard deviation
  • Min/max values
  • 95th and 99th percentile latency
  • Statistical sample count

3. Regression Detection

  • Compares current results against previous baseline
  • Configurable per-benchmark thresholds (default: 10%)
  • Automatic PR commenting on GitHub
  • Fails CI/CD if regressions detected

4. Data Persistence

  • Results stored in .benchmarks/ directory (git-ignored)
  • JSON format for programmatic analysis
  • Automatic cleanup (keeps last 10 runs)

5. Flexible CLI

Commands available:

  • compare - Check for regressions
  • report - Generate markdown report
  • export - Export results as JSON
  • list - List all stored benchmarks
  • clean - Remove old results

6. CI/CD Integration

GitHub Actions workflow provides:

  • Automatic runs on push/PR
  • Matrix testing across Node versions
  • Detailed PR comments
  • Artifact preservation
  • Main branch report generation

Usage Examples

Run all benchmarks

npm run bench

Run specific suite

npm run bench:event-decoding

Compare against baseline

npm run bench:compare

Output:

==================================================
EVENT-DECODING BENCHMARKS
==================================================

## Summary
- Total benchmarks: 3
- Regressions detected: 0
- Improvements: 1

✓ All benchmarks within threshold

Generate detailed report

npm run bench:report event-decoding

Watch mode for development

npm run bench:watch

Export results for analysis

ts-node src/benchmarks/cli.ts export api-responses ./result.json

Performance Targets

Critical paths are monitored with these targets:

Component Operation Target Threshold
Event Decoding SEP-41 transfer < 0.5ms 10%
Event Decoding Complex event < 1.0ms 15%
API Response List (50 items) < 100ms 15%
API Response Pagination < 200ms 20%
RPC Calls getEvents < 5ms 12%
RPC Calls Batch processing < 20ms 15%

CI/CD Behavior

When a PR is submitted:

  1. Workflow runs benchmarks automatically
  2. Results compared against main branch baseline
  3. PR comment posted with summary:
    • Total benchmarks run
    • Regressions detected
    • Improvements detected
  4. If regressions > threshold:
    • PR comment highlights each regression
    • Job fails (exit code 1)
    • Must be addressed before merge
  5. Artifacts uploaded for investigation

Storage

Results stored in .benchmarks/:

.benchmarks/
├── event-decoding-2026-07-28.json (4.23KB)
├── event-decoding-2026-07-27.json (3.89KB)
├── api-responses-2026-07-28.json (5.87KB)
├── rpc-calls-2026-07-28.json (6.12KB)
└── comparison-result.json (2.45KB)

Each result includes:

  • Timestamp
  • All metrics (mean, median, percentiles, etc.)
  • Number of samples
  • Configured threshold

Integration Points

Event Decoding

  • Benchmarks: src/indexer/decoder.ts:decodeEvent()
  • Represents: Soroban event parsing and human-readable text generation
  • Critical: Used for every event in the blockchain indexer

API Responses

  • Benchmarks: src/api/{transactions,events,contracts}.ts
  • Represents: HTTP endpoint latency and throughput
  • Critical: Direct impact on user experience

RPC Calls

  • Benchmarks: src/indexer/rpc.ts:fetchEvents() and related
  • Represents: Stellar RPC round-trip time and network overhead
  • Critical: Blocking operations that slow down indexing

Next Steps

  1. First run: Execute npm run bench to create initial baseline
  2. CI setup: Push to repository - GitHub Actions will run automatically
  3. Monitor: Check PR comments for regression alerts
  4. Iterate: Optimize code if regressions detected before merging

Benefits

✅ Automatic regression detection - Catch performance issues before they reach production
✅ Quantified metrics - Statistical data prevents guessing
✅ Easy to add - Template-based benchmark creation
✅ CI integration - No manual steps required
✅ Developer-friendly - Clear reports and actionable insights
✅ Production-ready - Supports multiple Node.js versions

Technical Notes

  • Uses Vitest framework (already in project dependencies)
  • High-resolution timing via Node.js perf_hooks
  • Garbage collection collection between warmup and measurement
  • Timeout protection to prevent infinite loops
  • Statistical outlier handling via percentile metrics
  • Thread-safe for parallel test execution