Valkey Performance Benchmark

A benchmarking tool for Valkey, an in-memory data store. Measures performance across different commits and configurations, including TLS and cluster modes.

Features

• Benchmarks Valkey server with various commands (SET, GET, RPUSH, etc.)
• Tests with different data sizes and pipeline configurations
• Supports TLS and cluster mode testing
• Handles automatic server setup and teardown
• Collects detailed performance metrics and generates reports
• Compares performance between different Valkey versions/commits
• Provides CPU pinning via taskset using configuration file settings
• Runs continuous benchmarking via GitHub Actions workflow
• Tracks commits and manages progress automatically
• Includes Grafana dashboards for visualizing performance metrics
• Search module performance testing with valkey search, adaptable to other modules
• Performance profiling with flamegraph generation via perf
• Regression detection comparing the latest benchmarked commits in PostgreSQL

Prerequisites

• Git
• Python 3.6+
• Linux environment (for taskset CPU pinning)
• Build tools required by Valkey (gcc, make, etc.)

Development Setup

Use a virtual environment for development:

# Create and activate venv
python3 -m venv venv
. venv/bin/activate

# Install dependencies
pip install --require-hashes -r requirements.txt

# Install pip-tools for dependency management
pip install pip-tools

Updating Dependencies

To update dependencies, edit requirements.in and regenerate the lock file:

. venv/bin/activate
pip-compile --generate-hashes requirements.in -o requirements.txt

Additional Prerequisites for Search Tests

• valkey-search module
• perf tool for profiling (optional: sudo yum install perf or sudo apt-get install linux-tools-generic)
• bunzip2 for dataset extraction (sudo yum install bzip2 or sudo apt-get install bzip2)

Project Structure

valkey-perf-benchmark/
├── .github/
│   ├── workflows/                    # GitHub Actions workflows
│   │   ├── valkey_benchmark.yml      # Continuous core benchmarking (self-hosted runners)
│   │   ├── search_benchmark.yml      # Continuous valkey-search module benchmarking
│   │   ├── benchmark_release_tags.yml # Benchmark specific Valkey release tags
│   │   ├── module-benchmark.yml      # Module framework smoke test (CI)
│   │   ├── basic.yml                 # Basic benchmark validation
│   │   ├── cluster_tls.yml           # Cluster and TLS specific tests
│   │   ├── tests.yml                 # Unit test suite (pytest)
│   │   ├── check_format.yml          # Code formatting checks (black)
│   │   └── sync-dashboards.yml       # Sync Grafana dashboards to the deployment
│   └── workflow-templates/
│       └── pr-benchmark-template.yml # Reusable PR-triggered benchmark template
├── configs/                  # Benchmark configuration files
│   ├── benchmark-configs.json           # Standard core benchmark configs
│   ├── benchmark-config-arm.json        # Core configs tuned for arm hosts
│   ├── benchmark-configs-cluster-tls.json # Cluster + TLS core configs
│   ├── benchmark-config-tag-arm.json    # Tag-index benchmark configs (arm)
│   ├── fts-benchmarks-arm.json          # Full search test suite (16 groups, arm)
│   ├── fts-benchmarks-shortened-arm.json # Shortened search suite (4 groups, arm)
│   └── module-test-arm.json             # Minimal module framework smoke test
├── dashboards/              # Grafana dashboards and AWS infrastructure
│   ├── grafana/            # Grafana dashboard definitions and Helm config
│   ├── kubernetes/         # Kubernetes manifests (ALB Ingress)
│   ├── infrastructure/     # CloudFormation templates
│   ├── scripts/            # Phase-based deployment scripts (00-06)
│   ├── schema.sql          # PostgreSQL database schema
│   └── README.md           # Deployment documentation
├── utils/                   # Utility scripts (Python package)
│   ├── compare_benchmark_results.py # Result comparison, statistics, and graphs
│   ├── postgres_track_commits.py    # Commit tracking and management (PostgreSQL)
│   ├── push_to_postgres.py          # Push metrics to PostgreSQL
│   ├── detect_regression.py         # Detect regressions between latest commits
│   ├── cpu_utils.py                 # CPU core range parsing and allocation
│   └── git_utils.py                 # Git ref resolution and commit fetching
├── benchmark.py             # Main entry point (core and modules)
├── valkey_build.py          # Builds Valkey server from source (ServerBuilder)
├── benchmark_build.py       # Builds valkey-benchmark from unstable (BenchmarkBuilder)
├── module_build.py          # Builds Valkey module .so files (ModuleBuilder)
├── valkey_server.py         # Manages Valkey server instances (ServerLauncher)
├── valkey_benchmark.py      # Client-side benchmark execution (ClientRunner) — core + modules
├── profiler.py              # Generic performance profiler (flamegraphs)
├── cpu_monitor.py           # CPU monitoring during tests
├── per_cpu_monitor.py       # Per-CPU monitoring (scheduler issue detection)
├── process_metrics.py       # Parses and formats benchmark results (MetricsProcessor)
├── tests/                   # Test suite
│   ├── integration/        # Integration tests (+ README)
│   └── test_*.py           # Unit tests (pytest)
├── scripts/                 # Helper scripts
│   └── setup_datasets.py   # Search dataset + query generator
├── datasets/                # Search test datasets and queries (auto-generated)
│   ├── search_terms.csv
│   └── proximity_phrases.csv
├── requirements.in          # Direct dependencies (human-editable)
└── requirements.txt         # Locked dependencies with hashes (auto-generated, includes test deps)

Each benchmark run clones a fresh copy of the Valkey repository for the target commit. When --valkey-path is omitted, the repository is cloned into valkey_<commit> and removed after the run to maintain build isolation and repeatability.

Usage

Basic Usage

# Run server and client benchmarks together with default configuration
python benchmark.py

# Run only the client component against an existing server
python benchmark.py --mode client --target-ip 192.168.1.100

# Use a specific configuration file
python benchmark.py --config ./configs/my-custom-config.json

# Benchmark specific commit(s) or ref(s)
python benchmark.py --commits 1a2b3c4d

# Use a pre-existing Valkey dir
python benchmark.py --valkey-path /path/to/valkey

# Without --valkey-path a directory named valkey_<commit> is cloned and later removed

# Use a custom valkey-benchmark executable
# (when omitted, the latest valkey-benchmark is cloned and built from unstable)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark

# Use a pre-running Valkey server (skips build / launch / cleanup)
python benchmark.py --valkey-path /path/to/valkey --use-running-server

--mode accepts both (default — run server and client on the same host) or client (run only the client against an existing server). There is no separate server-only mode.

Comparison Mode

# Compare against a baseline ref
python benchmark.py --commits HEAD --baseline unstable

# Run multiple benchmark runs for statistical reliability
python benchmark.py --commits HEAD --runs 5

Benchmark Comparison and Analysis

The project includes a comparison tool for analyzing benchmark results with statistical analysis and graph generation.

Compare Benchmark Results

# Basic comparison between two result files
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md

# Generate graphs along with comparison
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --graphs --graph-dir graphs/

# Filter to show only RPS metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics rps --graphs

# Filter to show only latency metrics
python utils/compare_benchmark_results.py --baseline results/commit1/metrics.json --new results/commit2/metrics.json --output comparison.md --metrics latency --graphs

Comparison Tool Features

• Automatic Run Averaging: Groups and averages multiple benchmark runs with identical configurations
• Statistical Analysis: Calculates means, standard deviations, and Coefficient of Variation (CV) with sample standard deviation (n-1)
• Coefficient of Variation: Provides normalized variability metrics (CV = σ/μ × 100%) for scale-independent comparison across performance metrics
• Graph Generation: matplotlib-based visualization including:
  • Consolidated comparison graphs for all metrics
  • Variance line graphs showing individual run values with error bars
  • RPS-focused filtering for integration purposes
• Metrics Filtering: Supports filtering by metric type (all, rps, latency)
• Standardized Output: Generates markdown reports with statistical information including CV
• Module Benchmark Support: Uses the module commit as the version identifier and reports the underlying core engine commit (see below)

Module Benchmark Comparisons

When comparing module benchmarks (e.g. valkey-search), each result record carries both a module_commit (the module build under test) and a commit (the core Valkey engine it ran against). The comparison tool handles these as follows:

• Version identifier: If module_commit is present it is used as the version shown in the report title, table headers, and graph legends. Otherwise the tool falls back to commit, then to the run timestamp.
• Core commit line: For module comparisons the report adds a **Core commit:** line so the core engine version is visible alongside the module version:
  • **Core commit:** <sha> when both sides ran against the same core commit
  • **Core commit:** <baseline-sha> (baseline) → <new-sha> (new) when they differ

Statistical Display Format

When multiple runs are available, the comparison tool displays comprehensive statistical information:

Metric Value (n=X, σ=Y, CV=Z%)

Where:

• n: Number of runs
• σ: Standard deviation
• CV: Coefficient of Variation as a percentage

The Coefficient of Variation (CV) is useful for:

• Scale-independent comparison: Compares variability across metrics with different units (e.g., RPS vs latency)
• Performance consistency assessment: Lower CV indicates more consistent performance
• Benchmark reliability evaluation: High CV indicates unstable test conditions

Interpreting Significance

• % Change: The relative difference between the two means (e.g., +3±2%). The ± is the uncertainty range — given the noise in the runs, the true change likely falls within that range (so +3±2% means somewhere between +1% and +5%).
• Significance indicators: Determined by whether the 95% confidence intervals of the baseline and new means overlap.
  • ✅ — CIs do not overlap, change is in the favorable direction (higher RPS or lower latency)
  • ❌ — CIs do not overlap, change is in the unfavorable direction
  • ➖ — CIs overlap, the difference cannot be distinguished from run-to-run noise
  • ❔ — One or both sides have only a single run (n ≤ 1), so no CI can be computed

A large % change may still be marked ➖: if variance is high — the indicator reflects statistical confidence, not the magnitude of the change.

Graph Types

1. Consolidated Comparison Graphs: Single graphs showing all metrics with legend format {commit}-P{pipeline}/IO{io_threads}
2. Variance Line Graphs: Individual run values with standard deviation visualization and error bars

Advanced Options

# Use an already running Valkey server (client mode only with `--valkey-path`)
python benchmark.py --mode client --valkey-path /path/to/valkey --use-running-server

# Specify custom results directory
python benchmark.py --results-dir ./my-results

# Set logging level
python benchmark.py --log-level DEBUG

# Use a custom valkey-benchmark executable (useful for testing modified versions)
python benchmark.py --valkey-benchmark-path /path/to/custom/valkey-benchmark

# Run only a subset of test groups / scenarios (test_groups configs)
python benchmark.py --config configs/fts-benchmarks-arm.json --module search --groups 1,2 --scenarios a,b

# Run a single cluster mode from a config that lists both [false, true]
python benchmark.py --config configs/fts-benchmarks-arm.json --cluster-mode-filter true

# Skip profiling passes and config_sets for a quick single pass
python benchmark.py --skip-profiling --skip-config-set

# Provide the repository so comparison reports link to commits
python benchmark.py --commits HEAD --baseline unstable --repository valkey-io/valkey

Benchmarking Remote Servers and Pre-Running Servers

When using --use-running-server or benchmarking remote servers, restarting the server between benchmark runs is the user's responsibility. Failure to restart between runs affects test results.

Custom Valkey-Benchmark Executable

The --valkey-benchmark-path option specifies a custom path to the valkey-benchmark executable. This is useful when:

• Testing a modified version of valkey-benchmark
• Using a pre-built binary from a different location
• Benchmarking with a specific version of the tool

When not specified, the tool clones and builds the latest valkey-benchmark from the Valkey unstable branch, so the benchmark client stays independent of the server commit under test.

# Example: Use a custom benchmark tool
python benchmark.py --valkey-benchmark-path /usr/local/bin/valkey-benchmark

# Example: Use with custom Valkey path
python benchmark.py --valkey-path /custom/valkey --valkey-benchmark-path /custom/valkey/src/valkey-benchmark

Configuration

Create benchmark configurations in JSON format. Each object represents a single set of options and configurations are not automatically cross-multiplied. Example:

[
  {
    "requests": [10000000],
    "keyspacelen": [10000000],
    "data_sizes": [16, 64, 256],
    "pipelines": [1, 10, 100],
    "commands": ["SET", "GET"],
    "cluster_mode": "yes",
    "tls_mode": "yes",
    "warmup": 10,
    "io-threads": [1, 4, 8],
    "server_cpu_range": "0-1",
    "client_cpu_range": "2-3",
    "custom-server-configs": {
      "maxmemory": "4gb",
      "timeout": "300"
    }
  }
]

Internally, this basic format is compiled into the scenario model at load time: each combination of the list-valued fields (requests x keyspacelen x data_sizes x pipelines x clients x commands) becomes a generated test group containing a single scenario, and the runner executes only the test_groups path. This is purely an internal translation: benchmark invocations and metrics.json outputs are unchanged, and both the basic format and the test_groups format (see Module Config Structure) remain fully supported.

Custom Server Configs Examples

Add server configs the benchmark does not manage (e.g. memory limits, timeouts):

"custom-server-configs": {
  "maxmemory": "4gb",
  "timeout": "300",
  "maxclients": "10000"
}

Combine with a baseline .conf file:

"custom-server-config-file": "/etc/valkey/baseline.conf",
"custom-server-configs": {
  "hz": "100",
  "tcp-keepalive": "60"
}

Configuration Parameters

Parameter	Description	Data Type	Multiple Values
requests	Number of requests to perform	Integer	Yes
keyspacelen	Key space size (number of distinct keys)	Integer	Yes
data_sizes	Size of data in bytes	Integer	Yes
pipelines	Number of commands to pipeline	Integer	Yes
clients	Number of concurrent client connections	Integer	Yes
commands	Valkey commands to benchmark	String	Yes
cluster_mode	Whether to enable cluster mode	String ("yes"/"no")	No
tls_mode	Whether to enable TLS	String ("yes"/"no")	No
warmup	Warmup time in seconds before benchmarking	Integer	No
io-threads	Number of I/O threads for server	Integer	Yes
server_cpu_range	CPU cores for server (e.g. "0-3", "0,2,4", or "144-191,48-95")	String	No
client_cpu_range	CPU cores for client (e.g. "4-7", "1,3,5", or "0-3,8-11")	String	No
custom-server-configs	Additional server configuration options the benchmark does not manage (e.g. {"maxmemory": "4gb", "timeout": "300"}).	Object (key-value pairs)	No
custom-server-config-file	Path to a Valkey-format .conf file passed positionally to valkey-server. Used as a baseline configuration.	String (path)	No

Note on custom-server-configs: This field lets you pass additional configuration options to the Valkey server at startup — settings the benchmark itself does not manage (e.g. maxmemory, timeout, maxclients, tcp-keepalive, hz).

Precedence: Configs are applied in this order on the valkey-server command line:

1. custom-server-config-file (positional, parsed first — lowest priority)
2. custom-server-configs (--key value flags)
3. Benchmark-managed defaults (--key value flags — highest priority via Valkey's last-wins semantics)

This means harness-critical settings (port, daemonize, logfile, cluster-*, etc.) always take effect regardless of what the user supplies. Setting them in custom-server-configs is allowed but has no effect.

When warmup is provided for read commands, the benchmark performs three stages:

1. A data injection pass using the corresponding write command with --sequential to seed the keyspace.
2. A warmup run of the read command (without --sequential) for the specified duration.
3. The main benchmark run of the read command.

Supported commands:

"SET", "GET", "RPUSH", "LPUSH", "LPOP", "SADD", "SPOP", "HSET", "GET", "MGET", "LRANGE", "SPOP", "ZPOPMIN"

Scenario-based Configurations (test_groups)

For module benchmarks (e.g. valkey-search) a scenario-based format is also supported. Instead of a cross-product of commands x data_sizes x ..., each scenario is an explicit test with its own command, dataset, clients, duration/requests, and warmup. Scenarios are grouped under test_groups. Sub-flags of the command can be expanded via options (e.g. NOCONTENT) into separate variants.

Two scenario types worth calling out:

• type: "write" / type: "read" — a single benchmark process against the server. Reads may be cluster_execution: "parallel" to fan out one client per cluster node.
• type: "mixed" — spawns concurrent write and read processes in the same test window. Each sub-scenario listed under writes: [...] and reads: [...] becomes its own valkey-benchmark process on its own CPU range and produces its own metric entry (test_phase: mixed_write / mixed_read). Options declared on a mixed scenario are applied to every read sub-scenario, not to a (non-existent) top-level command.

Datasets consumed by scenarios can be generated ahead of time from a dataset_generation block in the same config, via scripts/setup_datasets.py. Supported CSV transforms include wikipedia, inject, repeat, prefix_gen, proximity_phrase, expansion, fuzzy, numeric_range, and tag_list. The fuzzy transform pairs with a type: "fuzzy" query generator so a query term matches variant 0 of its corresponding dataset row and target_distance-edit variants (insert/delete/substitute) match under fuzzy search.

See configs/module-test-arm.json for a runnable example that includes a mixed workload and per-cluster-execution options.

Results

Benchmark results are stored in the results/ directory, organized by commit ID:

results/
└── <commit-id>/
    ├── logs.txt                         # Benchmark logs
    ├── metrics.json                     # Performance metrics in JSON format
    └── valkey_log_cluster_disabled.log  # Valkey server logs

Sample metrics.json

[
  {
    "timestamp": "2025-05-28T01:29:42+02:00",
    "commit": "ff7135836b5d9ccaa19d5dbaf2a0b0325755c8b4",
    "command": "SET",
    "data_size": 16,
    "pipeline": 10,
    "clients": 10000000,
    "requests": 10000000,
    "rps": 556204.44,
    "avg_latency_ms": 0.813,
    "min_latency_ms": 0.28,
    "p50_latency_ms": 0.775,
    "p95_latency_ms": 1.159,
    "p99_latency_ms": 1.407,
    "max_latency_ms": 2.463,
    "cluster_mode": false,
    "tls": false
  }
]

Continuous Benchmarking & CI/CD

GitHub Actions Workflows

The project includes several GitHub Actions workflows for automated testing and deployment:

• valkey_benchmark.yml: Continuous benchmarking workflow that runs on self-hosted EC2 runners
  • Benchmarks new commits from the Valkey unstable branch
  • Manages commit tracking via PostgreSQL
  • Uploads results to S3 and pushes metrics to PostgreSQL
  • Supports manual triggering with configurable commit limits
• search_benchmark.yml: Continuous valkey-search module benchmarking
  • Runs on a schedule (every 4 hours) and on manual dispatch
  • Determines core and module commits, builds valkey-server + valkey-search, and benchmarks
  • Configurable module branch (default main), repeat count, cluster mode, and profiling
• benchmark_release_tags.yml: Benchmarks a specified list of Valkey release tags
• module-benchmark.yml: Module framework smoke test on GitHub runners
  • Validates the module benchmarking framework with a minimal test
  • Uses configs/module-test-arm.json (small, quick)
  • Builds valkey-server + valkey-search and runs a quick smoke test
• basic.yml: Basic benchmark validation
• cluster_tls.yml: Tests for cluster and TLS configurations
• tests.yml: Runs the unit test suite (pytest)
• check_format.yml: Code formatting validation (black)
• sync-dashboards.yml: Syncs Grafana dashboard JSON changes to the deployment

Commit Tracking

The system uses PostgreSQL to track benchmarking progress. Commits are stored in the benchmark_commits table with their status, configuration, and architecture.

Status Descriptions

• in_progress: Workflow has selected the commit and is running the benchmark
• complete: Full workflow completed - metrics are in PostgreSQL and available in dashboards

Automatic Cleanup

The system cleans up in_progress entries on the next workflow run. This ensures:

• Failed benchmark runs are retried
• Commits stuck in progress do not block future runs
• The tracking reflects completed work accurately

Config Tracking

Each commit is tracked with the actual configuration content used for benchmarking. This provides:

• Comparison of results across different configurations by actual content
• Tracking of exact config parameters used for each benchmark
• Detection when config content changes even if file name stays the same
• Benchmarking the same commit with different configs
• Reproducibility by storing complete config data

Performance Dashboard

The project includes a complete AWS infrastructure for visualizing benchmark results using Grafana:

• AWS EKS Fargate - Serverless Kubernetes (no EC2 nodes)
• Amazon RDS PostgreSQL - Stores benchmark metrics and Grafana configuration
• CloudFront CDN - Global content delivery with HTTPS
• Application Load Balancer - Secured for CloudFront-only access

See dashboards/README.md for complete deployment guide and architecture details.

Utilities

Commit Management

• utils/postgres_track_commits.py: Manages commit tracking, status updates, and cleanup operations using PostgreSQL
• utils/push_to_postgres.py: Pushes benchmark metrics to PostgreSQL with dynamic schema support
• utils/compare_benchmark_results.py: Compares benchmark results across commits (statistics + graphs)
• utils/detect_regression.py: Detects performance regressions between the latest two benchmarked commits in PostgreSQL
• utils/cpu_utils.py: CPU core range parsing and per-node allocation for servers/clients
• utils/git_utils.py: Resolves git refs and fetches commits (handles shallow clones)

Configuration Files

• configs/benchmark-configs.json: Standard benchmark configurations
• configs/benchmark-configs-cluster-tls.json: Specialized configurations for cluster and TLS testing

Development

Local Development

For local development, simply run:

python benchmark.py

Code formatting is enforced by CI using black. To format locally:

pip install black==25.1.0
black .

Running Tests

The project includes a comprehensive test suite:

• Unit tests: Cover core logic functions (parsing, validation, statistics, metrics processing)
• Integration tests: Validate benchmark workflows with mock components

Tests run without requiring a Valkey server or PostgreSQL.

# Install dependencies (includes test deps)
pip install --require-hashes -r requirements.txt

# Run all tests
python -m pytest tests/ -v

# Run only unit tests
python -m pytest tests/ -v --ignore=tests/integration/

# Run only integration tests
python -m pytest tests/integration/ -v

# Run tests excluding slow tests
python -m pytest tests/ -v -m "not slow"

Integration Tests

The integration tests (tests/integration/) validate benchmark workflows end-to-end using mock components — no Valkey server, database, or network required. See tests/integration/README.md for details.

Adding New Configurations

Create new JSON configuration files in the configs/ directory following the existing format. Each configuration object represents a benchmark scenario.

Adapting the Framework for a New Module

The framework is module-agnostic: it never hard-codes anything about valkey-search. All modules run through the same path — benchmark.py builds and launches the server (loading the module .so), and ClientRunner in valkey_benchmark.py executes the scenarios described by the config's test_groups. --module is simply a label that routes results to results/{module}_tests/. No Python code changes are required to add a module.

To benchmark a new module (JSON, Bloom, TimeSeries, a custom module, …) you supply three things and reuse everything else (server lifecycle, CPU pinning, profiling, metrics, PostgreSQL upload, PR workflow):

1. Build the module .so with its native build system, and note the path:

   cd ../valkey-json && make      # or ./build.sh / cmake — whatever the module uses
   ls -lh .build-release/libjson.so

2. Author a config using the test_groups / scenarios structure (copy the shape of configs/fts-benchmarks-arm.json or the smaller configs/module-test-arm.json). Point the server at your module via the modules array and describe the workload:

   • setup_commands — your module's index/schema setup (search uses FT.CREATE; JSON might need none).
   • command — any command string your module accepts. Placeholders (__rand_int__, __field:NAME__, {tag}) work regardless of module.
   • dataset / dataset_generation — optional; add generated data if your workload needs it (the transforms in scripts/setup_datasets.py are generic and extensible).

3. Run it through the same entry point search uses:

   python benchmark.py \
     --module json \
     --module-path ../valkey-json/.build-release/libjson.so \
     --valkey-path ../valkey \
     --config configs/json-benchmarks.json \
     --groups 1

Results land in results/json_tests/ and can be compared with utils/compare_benchmark_results.py and pushed to PostgreSQL with utils/push_to_postgres.py exactly like search results. The generic infrastructure (profiler.py, cpu_monitor.py / per_cpu_monitor.py, process_metrics.py) applies automatically.

To wire a module into continuous CI benchmarking, see Onboarding a New Module to Continuous Benchmarking.

Module Testing (FTS, JSON, etc.)

The framework supports generic module testing through a unified ClientRunner.

Module Config Structure

Module tests use structured test_groups with scenarios:

{
  "test_groups": [{
    "group": 1,
    "scenarios": [
      {
        "type": "write",
        "cluster_execution": "single",
        "setup_commands": ["FT.CREATE idx ..."],
        "command": "HSET ...",
        "dataset": "data.xml",
        "clients": 1000,
        "maxdocs": 50000
      },
      {
        "type": "read",
        "cluster_execution": "parallel",
        "command": "FT.SEARCH idx __field:term__",
        "dataset": "queries.csv",
        "clients": 1000,
        "duration": 60,
        "warmup": 20
      }
    ]
  }],
  "cluster_mode": false,
  "tls_mode": false
}

Scenario fields: Each scenario names its workload with exactly one of test or command (except type: mixed scenarios, which use writes/reads sub-scenarios instead):

Field	Description
test	Predefined valkey-benchmark test name, run as -t NAME. Used by compiled basic-format configs.
command	Arbitrary command line, run after --. Supports __rand_int__ placeholders.
data_size	Payload size in bytes, passed as -d N when present.
keyspacelen	Per-scenario key space size, passed as -r N. Falls back to the config-level keyspacelen[0].
warmup_inline	Adds --warmup N to the main benchmark run. Distinct from warmup, which performs a separate warm-up run first.
restart_before	Restarts the managed server before the scenario runs (flushes the database instead when using a running server).
populate_with	Write workload that seeds the keyspace before a read test. For a test scenario it is a predefined write name (e.g. SET), run as -t NAME; for a command scenario it is an arbitrary write command string (e.g. SET key:__rand_int__ __data__), run after --. The populate pass runs sequentially and shares the main run's seed, so the read hits the seeded keys.

Cluster Mode Support

Multi-node clusters: Config with cluster_mode array runs both single-node and distributed tests:

{
  "cluster_mode": [false, true],
  "cluster_nodes": 5,
  "cluster_ports": [6379, 6380, 6381, 6382, 6383],
  "cpu_allocation": {
    "cores_per_server": 8,
    "cores_per_client": 8,
    "servers": ["0-7", "8-15", "16-23", "24-31", "32-39"],
    "clients": ["40-47", "48-55", "56-63", "64-71", "72-79"]
  },
  "modules": [
    {
      "path": "../valkey-search/.build-release/libsearch.so",
      "startup_args": ["--use-coordinator"]
    }
  ]
}

Module Loading: Supports loading multiple modules with per-module startup arguments:

{
  "modules": [
    {
      "path": "../valkey-search/.build-release/libsearch.so",
      "startup_args": ["--use-coordinator", "--timeout", "30"]
    },
    {
      "path": "../valkey-json/.build-release/libjson.so",
      "startup_args": []
    }
  ]
}

CPU Allocation: Two methods (mutually exclusive):

1. New (recommended): cpu_allocation object

   • Automatic: Provide cores_per_server + cores_per_client

     "cpu_allocation": {
       "cores_per_server": 8,
       "cores_per_client": 8
     }

     Framework calculates ranges: servers [0-7, 8-15, 16-23...], clients [40-47, 48-55...]

   • Manual override: Provide servers + clients arrays

     "cpu_allocation": {
       "servers": ["0-7", "8-15", "16-23", "24-31", "32-39"],
       "clients": ["40-47", "48-55", "56-63", "64-71", "72-79"]
     }

     Explicit ranges used as-is (cores_per_* ignored if present)

2. Old: server_cpu_range + client_cpu_range (single-node only)

CME Parallel Execution: Scenarios can specify execution strategy:

• "cluster_execution": "single" - One client with cluster routing (default)
• "cluster_execution": "parallel" - N clients (one per node), aggregated metrics
• Optional: "parallel_clients": 10 - Custom client count

Filter modes: --cluster-mode-filter [false|true] runs specific mode only.
Key pattern: Use {tag} in HSET for cluster routing: HSET rd0-{tag}:__rand_int__

Onboarding a New Module to Continuous Benchmarking

A module team (valkey-json, valkey-bloom, a custom module, …) can reuse this framework's continuous benchmarking workflow without forking it. The reference implementation is .github/workflows/search_benchmark.yml: it runs on a schedule (and manual dispatch), benchmarks new module commits over time, stores results in PostgreSQL, and surfaces them in Grafana. Model your module's workflow on it.

What the workflow does, step by step (adapt each to your module):

1. Checks out the module repo, valkey core (unstable), and this benchmark repo.
2. Builds valkey-server, the valkey-benchmark client, and the module .so (search uses ./build.sh → .build-release/libsearch.so; use your module's build system and .so path).
3. Determines which module commits still need benchmarking via utils/postgres_track_commits.py determine and marks them in_progress.
4. For each commit: checkout + build the module, then run python benchmark.py --module <name> --module-path <.so> --valkey-path <core> --config <config> ....
5. Pushes results to PostgreSQL with utils/push_to_postgres.py and marks the commits complete.

Infrastructure it needs:

• A self-hosted runner (the search workflow uses [self-hosted, valkey-search-arm]) sized for your benchmark, with build tools and perf installed.
• PostgreSQL + AWS creds (RDS with IAM auth in the search setup) provided as repo secrets.
• A config using the test_groups / scenarios structure with your module's setup_commands, command templates, and datasets — see Adapting the Framework for a New Module.
• A Grafana dashboard (optional) — add a JSON under dashboards/grafana/ and it is picked up by sync-dashboards.yml.

PostgreSQL Table Selection (--table)

Both utils/push_to_postgres.py and utils/postgres_track_commits.py take a --table identifier that decides which metrics table your results go into — this is how each module keeps its data separate. resolve_table_name() maps it:

--table value	Resulting table
core (default)	benchmark_metrics
tag	benchmark_tags_metrics
any other id (e.g. search, json)	benchmark_metrics_{table} (e.g. benchmark_metrics_search)

The identifier must match ^[a-z][a-z0-9_]{0,30}$. So a new module typically passes its own name, e.g. --table json → results land in benchmark_metrics_json.

A separate --test-type flag (e.g. core, fts) is not used for table naming — it is stored on each row as a tag for filtering/grouping in dashboards.

python utils/push_to_postgres.py \
  --results-dir ./results/<core_sha>_<module_sha> \
  --table json --test-type json \
  --host <rds-endpoint> --port 5432 \
  --database <db> --username <user> --password "$PGPASSWORD"

Use the same --table value in postgres_track_commits.py (for commit tracking) and push_to_postgres.py (for results) so tracking and metrics stay aligned.

A module that only wants local/manual runs needs none of this CI infrastructure — just build the .so and run benchmark.py as shown in Running Module Tests Locally.

Setting Up PR Benchmarks

Both valkey core and module repositories can set up automated PR benchmarking using our unified workflow template.

Quick Start

1. Copy the template from this repo:

   cp .github/workflow-templates/pr-benchmark-template.yml \
      .github/workflows/benchmark-on-label.yml

2. Customize for your repository:

   • Runner label: Update runs-on with your self-hosted runner label
   • For module repos: Update MODULE_NAME, .so path, and build commands
   • For core repo: Remove or comment out module-specific steps

3. Trigger benchmarks by adding the run-benchmark label to any PR

Template Customization Guide

The template includes clear CUSTOMIZE markers for:

Module Repositories (valkey-search, valkey-json, etc.):

• Module build command (build.sh, make, cmake)
• Path to .so file (e.g., .build-release/libsearch.so)
• Module name for --module parameter
• Benchmark config file path

Core Repository (valkey/valkey):

• Uses conditional steps based on github.repository
• Most sections work without modification

See .github/workflow-templates/pr-benchmark-template.yml for detailed inline documentation.

Workflow Features

• Triggered by run-benchmark label on PRs
• Compares PR branch against base branch
• Posts results as PR comment
• Uploads artifacts for detailed analysis
• Automatic cleanup and label removal

Running Module Tests Locally

--module-path requires a pre-built .so file (not source directory) since modules use different build systems (make, cmake, build.sh) and may need specific compilers.

Build module first:

cd valkey-search
make BUILD_TLS=yes  # or ./build.sh, cmake, etc.
ls -lh .build-release/libsearch.so

Run benchmarks:

python benchmark.py \
  --module search \
  --module-path ../valkey-search/.build-release/libsearch.so \
  --valkey-path ../valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

Framework manages server lifecycle automatically.

Running FTS Tests

Note: Datasets are automatically generated on first run if missing. The initial run may take 30-60 minutes to download Wikipedia and generate datasets. Subsequent runs use cached datasets and start immediately.

Unified Entry Point (Recommended)

Use benchmark.py with --module search:

# Run FTS test Group 1 (datasets auto-generated if missing)
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

# Run specific scenarios within groups
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --scenarios a,b

# Filter by groups (works with any config using test_groups structure)
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1,2

Results saved to results/search_tests/ with optional flamegraphs if profiling enabled in config.

Note: --groups and --scenarios work with any configuration using the test_groups structure (core or module tests).

CPU Pinning Configuration

CPU pinning is configured in the config file:

{
  "server_cpu_range": "0-7",   // Pin server to cores 0-7 (when using --mode both)
  "client_cpu_range": "8-15"   // Pin benchmark client to cores 8-15
}

With --mode both (recommended): Framework manages server automatically with CPU pinning from config.

python benchmark.py \
  --module search \
  --module-path ../valkey-search/.build-release/libsearch.so \
  --valkey-path ../valkey \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

With --use-running-server (manual server management): Start server yourself with desired CPU pinning.

# Start server manually
taskset -c 0-7 /path/to/valkey-server --loadmodule libsearch.so ...

# Run benchmarks
python benchmark.py \
  --module search \
  --valkey-path /path/to/valkey \
  --use-running-server \
  --config configs/fts-benchmarks-arm.json \
  --groups 1

Dataset Generation System

The framework uses a transform-based dataset generation system that supports multiple testing strategies:

Config Format:

"dataset_generation": {
  "dataset_name.xml": {
    "doc_count": 50000,
    "fields": [
      {
        "name": "field0",
        "size": 100,
        "transforms": [
          {"type": "wikipedia"},
          {"type": "inject", "term": "MARKER_TERM", "percentage": 0.5}
        ]
      }
    ]
  }
}

Supported Transforms:

• wikipedia: Extract Wikipedia content (base text)
• inject: Add marker terms at specified percentage
• repeat: Duplicate terms N times (for term_repetition tests)
• prefix_gen: Generate prefix variations (for prefix_explosion tests)
• proximity_phrase: Generate N-term phrases for proximity testing
  • Parameters: term_count, combinations (1=best case, 100=worst case), repeats (copies per pattern)
  • Supports CSV output (no Wikipedia needed)
• expansion: Generate wildcard-expansion variants (term001_a, term001_aa, ...) grouped as expansion_count × docs_per_expansion copies × term_count base terms
• fuzzy: Generate misspelled variants for fuzzy-match tests
  • Parameters: variant_count, docs_per_variant, term_count, min_word_length, max_word_length, target_distance
  • Variant 0 is the correctly-spelled base word (matches type: "fuzzy" queries); variants ≥1 apply target_distance Levenshtein edits (insert/delete/substitute)
• numeric_range: Random numeric value in [min, max] (for NUMERIC index tests)
• tag_list: Pipe-separated random tags from a supplied list (for TAG index tests)
• vector: Marks a vector field. Vector data lives in a structured .npy file (not CSV); the presence of this transform routes generation to the structured NPY writer. Used for KNN / HNSW vector-search tests, including hybrid configs that combine a vector field with text/numeric/tag fields.

Compact Format (for field explosion):

"field_explosion_50k.xml": {
  "doc_count": 50000,
  "generate_fields": {
    "count": 50,
    "size": 1000,
    "transforms": [{"type": "wikipedia"}]
  }
}

Query Generation:

"query_generation": {
  "proximity_5term_queries.csv": {
    "type": "proximity_phrase",
    "doc_count": 100,
    "term_count": 5
  }
}

• Auto-generates query CSVs matching ingestion datasets
• Supports type-based generation (extensible for future query types)

Supported query types:

• proximity_phrase — multi-column phrase terms matching a proximity_phrase transform
• prefix / suffix — substring queries extracted from an existing source CSV
• expansion — zero-padded base terms (e.g. term001) matching an expansion transform
• fuzzy — deterministic base words matching variant 0 of a fuzzy transform, so every query has known matches in the dataset (parameters: min_word_length, max_word_length)
• tag_only — rotates through a supplied tags list to produce category filters for composed TAG queries
• vector — generates a structured .npy of (search_term, query_vector) pairs (L2-normalized random vectors, filename-seeded for reproducibility) plus a companion CSV of the search terms, for KNN / hybrid vector queries. Parameter: dimensions (default 256)

Search Test Groups

configs/fts-benchmarks-arm.json (the FTS Performance Benchmarks suite) defines the full search suite of 16 groups. Groups are selected with --groups and individual scenarios within a group with --scenarios. Each group begins with a write scenario that ingests its dataset, followed by the read (and occasionally mixed) scenarios that query it.

Group	Focus
1	Multi-field comprehensive (STEM enabled)
2	Proximity queries — 5-term best case (1 combination)
3	Proximity queries — 5-term worst case (100 combinations)
4	Proximity queries — 25-term worst case (100 combinations)
5	Prefix (wildcard) expansion — best case
6	Prefix (wildcard) expansion — worst case
7	Suffix expansion — best case
8	Suffix expansion — worst case
9	Hybrid queries (TEXT + NUMERIC + TAG)
10	Fuzzy matching — best case (distance 1)
11	Fuzzy matching — worst case (distance 3)
12	Posting tests — position map partitions and byte size
13	Radix tests — node growth and prefix locality
14	Misc tests — string intern, schema options, extreme case
15	Vector + text hybrid (KNN with filters)
16	Composed queries — nominal case (entries fetcher test)

Shortened Search Suite

configs/fts-benchmarks-shortened-arm.json (the FTS Shortened Performance Benchmarks suite) is a smaller, CI-sized version of the full suite. It is the default config for the continuous search_benchmark.yml workflow and is sized to finish quickly while still exercising every major search path (text, hybrid, and both vector index types), across 4 groups:

Group	Focus
P1	Text single-field baseline
P2	Multi-index hybrid (TEXT + NUMERIC + TAG)
P3	Vector FLAT (prefilter path)
P4	Vector HNSW (UsePreFiltering ratio branch)

Search Results

Results are saved to results/search_tests/:

• metrics.json - Performance metrics
• flamegraphs/ - Profiling data (if enabled)

Sample metrics structure (module runs add test_id, group/scenario, their descriptions, and — when supplied — module_commit / module_commit_timestamp):

{
  "test_id": "1_b",
  "test_phase": "read",
  "group": 1,
  "scenario": "b",
  "group_description": "Multi-field comprehensive (STEM enabled)",
  "scenario_description": "Single term all fields",
  "status": "success",
  "rps": 6596.42,
  "avg_latency_ms": 7.535,
  "p50_latency_ms": 5.367,
  "p95_latency_ms": 20.975,
  "cpu_avg_percent": 692.84,
  "cpu_peak_percent": 751.10,
  "memory_mb": 4469.87,
  "dataset": "field_explosion_50k.xml",
  "module_commit": "..."
}

test_id is {group}_{scenario} and test_phase is the scenario type (read, write, or mixed_read / mixed_write for mixed scenarios).

Performance Profiling

The framework includes a generic profiler that works with both core and FTS tests.

Attribution

This framework uses The FlameGraph project by Brendan Gregg for performance visualization. FlameGraph is licensed under CDDL 1.0 (Common Development and Distribution License). The FlameGraph scripts are automatically downloaded from the source repository when profiling is enabled.

Using Profiler in Core Tests

The PerformanceProfiler class can be integrated into any benchmark script:

from profiler import PerformanceProfiler

# Initialize profiler
profiler = PerformanceProfiler(results_dir, enabled=True)

# Example from search module:
profiler.start_profiling("search_1a", target_process="valkey-server")

# Run your benchmark
runner.run_benchmark_config()

# After benchmark completes
profiler.stop_profiling("search_1a")
# → Generates:
#    - flamegraph: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.svg
#    - perf report: results_dir/commit_id/flamegraphs/search_1a_20251218_080245_report.txt
#    - raw data: results_dir/commit_id/flamegraphs/search_1a_20251218_080245.perf.data

Profiler Features

• Flamegraph generation: Visual call stack analysis
• Auto-downloads scripts: Fetches flamegraph tools from GitHub on first use
• Profiling modes: cpu (cycles) or wall-time (all execution time)
• Function hotspot analysis: Identify CPU-intensive code paths
• Kernel + user space profiling: Complete stack traces with DWARF
• Generic implementation: Works with any process (valkey-server, redis-server, etc.)
• Configurable sampling: 999Hz default

profiling_sets Configuration

Run tests with multiple profiling configurations:

{
  "profiling_sets": [
    {"enabled": false},
    {"enabled": true, "mode": "wall-time", "sampling_freq": 999}
  ],
  "config_sets": [
    {"search.reader-threads": 1},
    {"search.reader-threads": 8}
  ]
}

Behavior:

• Iterates profiling_sets × config_sets
• Profiling OFF → Collects metrics in metrics.json
• Profiling ON → Generates flamegraphs, skips metrics
• Flamegraphs: group{X}_{scenario}_{config_values}_{timestamp}.svg

Per-scenario override:

{"id": "a", "profiling": {"delays": {"search": {"delay": 0, "duration": 10}}}}

Scenario overrides any profiling_set values.

Delay strategy pattern:

{
  "profiling_sets": [{
    "enabled": true,
    "delays": {
      "write": {"delay": 0, "duration": 10},
      "read": {"delay": 30, "duration": 10}
    }
  }],
  "scenarios": [{
    "id": "a",
    "type": "write",
    "profiling": {"delays": {"write": {"delay": 10, "duration": 10}}}
  }]
}

Group 1 write scenario uses 10s delay (dataset loading), others use 0s (immediate).

License

Please see the LICENSE.md