Skip to content

Commit 9e6276a

Browse files
ikolomiikolomi
committed
feat: add Ruby benchmark engine with full Java test parity
Add complete Ruby implementation of the resp-bench benchmark suite using the redis-rb client library. The Ruby engine mirrors the Java reference implementation architecture and produces identical NDJSON metrics output. Ruby engine (new): - BenchmarkEngine with hybrid multi-process + multi-threaded concurrency - SizedQueue-based backpressure for throughput scaling - Java-compatible LCG key generator (identical sequences with same seed) - Leaky bucket rate limiter (RPS/CPS limits) - HdrHistogram-based metrics with NDJSON output - RecordingClient for deterministic testing (error rates, log-normal latency) - Commands: GET, SET, PING Test suite (34 unit + 41 integration tests): - Unit: ConfigLoader, KeyGenerator, JavaRandom, RateLimiter - Integration: MetricsOutput (NDJSON format, latency p50 validation, long-tail histogram accuracy), ErrorMetrics (error rates, per-command tracking, connection errors), RecordingClient workloads, rate limiting - Full parity with Java test coverage; known differences documented (HDR encode not supported by Ruby gem, redis-rb lazy connections) Infrastructure: - Driver configs: example-redis-rb-standalone.json, reference/redis-rb.json - Makefile targets: ruby-build, ruby-test, ruby-run, ruby-info - CI workflow updated to include Ruby benchmarks - README updated with Ruby quick start and test parity table Signed-off-by: ikolomi <ikolomin@amazon.com>
1 parent 16066b7 commit 9e6276a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

48 files changed

+6067
-6
lines changed

‎.github/benchmark-config/drivers.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,8 @@
99
"configs/drivers/reference/spring-data-valkey-jedis-standalone.json",
1010
"configs/drivers/reference/spring-data-valkey-lettuce-standalone.json",
1111
"configs/drivers/reference/valkey-glide-standalone.json"
12+
],
13+
"ruby": [
14+
"configs/drivers/reference/redis-rb.json"
1215
]
1316
}

‎.github/workflows/benchmark.yml‎

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ on:
44
workflow_dispatch:
55

66
jobs:
7-
benchmark:
7+
benchmark-java:
88
runs-on: ubuntu-latest
99

1010
strategy:
@@ -81,12 +81,80 @@ jobs:
8181
- name: Upload results
8282
uses: actions/upload-artifact@v4
8383
with:
84-
name: benchmark-${{ steps.names.outputs.driver_name }}-${{ steps.names.outputs.workload_name }}
84+
name: benchmark-java-${{ steps.names.outputs.driver_name }}-${{ steps.names.outputs.workload_name }}
85+
path: ${{ steps.names.outputs.result_file }}
86+
retention-days: 30
87+
88+
benchmark-ruby:
89+
runs-on: ubuntu-latest
90+
91+
strategy:
92+
fail-fast: false
93+
matrix:
94+
driver:
95+
- configs/drivers/reference/redis-rb.json
96+
workload:
97+
- configs/workloads/reference/basic-standalone-single-client.json
98+
- configs/workloads/reference/basic-standalone-100-clients.json
99+
100+
steps:
101+
- uses: actions/checkout@v4
102+
103+
- name: Set up Ruby
104+
uses: ruby/setup-ruby@v1
105+
with:
106+
ruby-version: '3.2'
107+
bundler-cache: true
108+
working-directory: ruby
109+
110+
- name: Install build dependencies
111+
run: |
112+
sudo apt-get update
113+
sudo apt-get install -y build-essential
114+
115+
- name: Start Valkey server
116+
run: |
117+
# Use Makefile target which builds from source and configures with persistence disabled
118+
make server-standalone-start
119+
# Wait for server to be ready
120+
sleep 2
121+
# Verify server is up and persistence is disabled
122+
work/valkey/bin/valkey-cli ping
123+
work/valkey/bin/valkey-cli CONFIG GET save
124+
125+
- name: Extract names for result file
126+
id: names
127+
run: |
128+
DRIVER_NAME=$(basename ${{ matrix.driver }} .json)
129+
WORKLOAD_NAME=$(basename ${{ matrix.workload }} .json)
130+
echo "driver_name=$DRIVER_NAME" >> $GITHUB_OUTPUT
131+
echo "workload_name=$WORKLOAD_NAME" >> $GITHUB_OUTPUT
132+
echo "result_file=results/github-runner/reference/${DRIVER_NAME}-${WORKLOAD_NAME}.ndjson" >> $GITHUB_OUTPUT
133+
134+
- name: Run benchmark
135+
run: |
136+
mkdir -p results/github-runner/reference
137+
cd ruby && bundle exec ruby bin/resp-bench \
138+
--server localhost:6379 \
139+
--driver ../${{ matrix.driver }} \
140+
--workload ../${{ matrix.workload }} \
141+
--metrics ../${{ steps.names.outputs.result_file }} \
142+
--commit-id ${{ github.sha }}
143+
144+
- name: Stop Valkey server
145+
if: always()
146+
run: |
147+
make server-standalone-stop || true
148+
149+
- name: Upload results
150+
uses: actions/upload-artifact@v4
151+
with:
152+
name: benchmark-ruby-${{ steps.names.outputs.driver_name }}-${{ steps.names.outputs.workload_name }}
85153
path: ${{ steps.names.outputs.result_file }}
86154
retention-days: 30
87155

88156
generate-graphs:
89-
needs: benchmark
157+
needs: [benchmark-java, benchmark-ruby]
90158
runs-on: ubuntu-latest
91159
permissions:
92160
contents: write

‎Makefile‎

Lines changed: 41 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ WORK_DIR=$(shell pwd)/work
3232
server-start server-stop \
3333
java-build java-test java-run java-clean \
3434
python-build python-test python-run python-clean \
35+
ruby-build ruby-test ruby-run ruby-clean ruby-info \
3536
config-editor-build config-editor-dev
3637

3738
# ============================================================================
@@ -59,6 +60,12 @@ help:
5960
@echo " make python-test Run Python tests"
6061
@echo " make python-run Run Python benchmark"
6162
@echo ""
63+
@echo "Ruby Engine:"
64+
@echo " make ruby-build Install Ruby dependencies"
65+
@echo " make ruby-test Run Ruby tests"
66+
@echo " make ruby-run Run Ruby benchmark (requires DRIVER and WORKLOAD)"
67+
@echo " make ruby-info Show supported Ruby drivers and commands"
68+
@echo ""
6269
@echo "Config Editor:"
6370
@echo " make config-editor-build Build config editor UI"
6471
@echo " make config-editor-dev Run config editor in development mode"
@@ -309,6 +316,37 @@ python-clean:
309316
@echo "Python engine not yet implemented"
310317
@echo "Placeholder for: cd python && rm -rf __pycache__ *.egg-info dist build"
311318

319+
# ============================================================================
320+
# Ruby Engine
321+
# ============================================================================
322+
323+
ruby-build:
324+
cd ruby && bundle install
325+
326+
ruby-test:
327+
cd ruby && bundle exec rake test
328+
329+
ruby-unit-test:
330+
cd ruby && bundle exec rake unit
331+
332+
ruby-integration-test: server-standalone-start
333+
sleep 1
334+
cd ruby && VALKEY_HOST=localhost VALKEY_PORT=6379 bundle exec rake integration
335+
$(MAKE) server-standalone-stop
336+
337+
ruby-run: ruby-build
338+
cd ruby && bundle exec ruby bin/resp-bench \
339+
--server $(SERVER) \
340+
--driver ../$(DRIVER) \
341+
--workload ../$(WORKLOAD) \
342+
--metrics ../$(METRICS_OUTPUT)
343+
344+
ruby-clean:
345+
cd ruby && rm -rf vendor .bundle Gemfile.lock
346+
347+
ruby-info: ruby-build
348+
cd ruby && bundle exec ruby bin/resp-bench --info
349+
312350
# ============================================================================
313351
# Config Editor
314352
# ============================================================================
@@ -323,8 +361,8 @@ config-editor-dev:
323361
# All Languages
324362
# ============================================================================
325363

326-
build-all: java-build python-build
364+
build-all: java-build ruby-build python-build
327365

328-
test-all: java-test python-test
366+
test-all: java-test ruby-test python-test
329367

330-
clean-all: java-clean python-clean clean
368+
clean-all: java-clean ruby-clean python-clean clean

‎README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@ The graphs below show performance comparisons of Java client libraries running o
5555
| Language | Status | Supported Drivers |
5656
|----------|--------|-------------------|
5757
| Java | ✅ Ready | Jedis, Lettuce, Valkey-Glide, Redisson, Spring Data Valkey/Redis |
58+
| Ruby | ✅ Ready | redis-rb |
5859
| Python | 🚧 Planned | redis-py, aioredis, valkey-glide |
5960
| Go | 📋 Future | go-redis, rueidis |
6061
| Node.js | 📋 Future | ioredis, node-redis |
@@ -85,6 +86,13 @@ make java-run \
8586
WORKLOAD=configs/workloads/example-workload.json
8687
```
8788

89+
**Ruby:**
90+
```bash
91+
make ruby-run \
92+
DRIVER=configs/drivers/example-redis-rb-standalone.json \
93+
WORKLOAD=configs/workloads/example-workload.json
94+
```
95+
8896
**Python (when available):**
8997
```bash
9098
make python-run \
@@ -111,6 +119,7 @@ resp-bench/
111119
│ └── workloads/ # Workload definitions
112120
├── config-editor/ # React-based configuration editor UI
113121
├── java/ # Java benchmark engine
122+
├── ruby/ # Ruby benchmark engine
114123
├── python/ # Python benchmark engine (planned)
115124
├── docs/ # Documentation
116125
│ ├── ARCHITECTURE.md # System architecture
@@ -272,6 +281,15 @@ The script aggregates results using a 3-tuple key: `(commit_id, primary_driver_v
272281
| `make java-run` | Run benchmark (uses DRIVER, WORKLOAD, SERVER vars) |
273282
| `make java-info` | Show supported drivers and commands |
274283

284+
### Ruby Engine
285+
286+
| Target | Description |
287+
|--------|-------------|
288+
| `make ruby-build` | Install Ruby dependencies (bundle install) |
289+
| `make ruby-test` | Run unit and integration tests |
290+
| `make ruby-run` | Run benchmark (uses DRIVER, WORKLOAD, SERVER vars) |
291+
| `make ruby-info` | Show supported drivers and commands |
292+
275293
### Config Editor
276294

277295
| Target | Description |

‎configs/drivers/example-redis-rb-standalone.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"schema_version": "1.0",
3+
"description": "redis-rb client - standalone mode",
4+
"driver_id": "redis-rb",
5+
"mode": "standalone",
6+
"specific_driver_config": {}
7+
}

‎configs/drivers/reference/redis-rb.json‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
{
2+
"schema_version": "1.0",
3+
"description": "redis-rb client - reference configuration",
4+
"driver_id": "redis-rb",
5+
"mode": "standalone",
6+
"specific_driver_config": {}
7+
}

‎java/src/main/java/io/valkey/javabenchmark/engine/BenchmarkEngine.java‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,9 @@ public class BenchmarkEngine {
4040

4141
/** Default pipeline depth (max in-flight requests per client) */
4242
private static final int DEFAULT_PIPELINE_DEPTH = 1;
43+
44+
/** Progress logging interval in seconds */
45+
private static final int PROGRESS_LOG_INTERVAL_SECONDS = 10;
4346

4447
private final String host;
4548
private final int port;
@@ -193,6 +196,10 @@ private String executeWorkload(PhaseConfig phase, List<ClientSlot> clientSlots,
193196

194197
metrics.start();
195198

199+
// Progress logging state
200+
long lastLogTime = System.currentTimeMillis();
201+
long startTime = System.currentTimeMillis();
202+
196203
try {
197204
if (completion.isDurationBased()) {
198205
// Duration-based completion
@@ -201,6 +208,7 @@ private String executeWorkload(PhaseConfig phase, List<ClientSlot> clientSlots,
201208
while (System.currentTimeMillis() < endTime && running.get()) {
202209
submitRequestWithBackpressure(clientSlots, selector, keyGenerator, rateLimiter,
203210
metrics, requestCount, pendingCount, globalSlots);
211+
lastLogTime = logProgressIfNeeded(requestCount.get(), -1, lastLogTime, startTime);
204212
}
205213

206214
} else {
@@ -210,6 +218,7 @@ private String executeWorkload(PhaseConfig phase, List<ClientSlot> clientSlots,
210218
while (requestCount.get() < targetRequests && running.get()) {
211219
submitRequestWithBackpressure(clientSlots, selector, keyGenerator, rateLimiter,
212220
metrics, requestCount, pendingCount, globalSlots);
221+
lastLogTime = logProgressIfNeeded(requestCount.get(), targetRequests, lastLogTime, startTime);
213222
}
214223
}
215224

@@ -391,6 +400,35 @@ private void submitWarmup(List<ClientSlot> clientSlots, int warmupRequestsPerCli
391400

392401
logger.info("Warmup submitted, proceeding to measured workload (slots occupied by warmup will pace requests)");
393402
}
403+
404+
/**
405+
* Log progress if enough time has elapsed since last log.
406+
*
407+
* @param current current request count
408+
* @param target target request count (-1 for duration-based completion)
409+
* @param lastLogTime timestamp of last log
410+
* @param startTime timestamp when workload started
411+
* @return updated lastLogTime
412+
*/
413+
private long logProgressIfNeeded(long current, long target, long lastLogTime, long startTime) {
414+
long now = System.currentTimeMillis();
415+
if (now - lastLogTime < PROGRESS_LOG_INTERVAL_SECONDS * 1000) {
416+
return lastLogTime;
417+
}
418+
419+
long elapsedMs = now - startTime;
420+
long rate = elapsedMs > 0 ? (current * 1000 / elapsedMs) : 0;
421+
422+
if (target > 0) {
423+
double percent = (current * 100.0 / target);
424+
logger.info("Progress: {}/{} requests ({} %) - {} req/s",
425+
current, target, String.format("%.1f", percent), rate);
426+
} else {
427+
logger.info("Progress: {} requests - {} req/s", current, rate);
428+
}
429+
430+
return now;
431+
}
394432

395433
/**
396434
* Wraps a BenchmarkClient with semaphore-based backpressure control.

‎ruby/Gemfile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# frozen_string_literal: true
2+
3+
source "https://rubygems.org"
4+
5+
gemspec
6+
7+
# Core dependencies
8+
gem "redis", "~> 5.0" # redis-rb client
9+
gem "HDRHistogram", "~> 0.1" # HdrHistogram for latency metrics
10+
gem "oj", "~> 3.16" # Fast JSON serialization
11+
gem "concurrent-ruby", "~> 1.2" # Thread-safe data structures
12+
13+
# Optional: valkey-glide-ruby (uncomment when needed)
14+
# gem "valkey"
15+
16+
group :development, :test do
17+
gem "minitest", "~> 5.0"
18+
gem "rake", "~> 13.0"
19+
gem "rubocop", "~> 1.50"
20+
end

0 commit comments

Comments
 (0)