Skip to content

Commit 33b6f48

Browse files
ikolomiikolomi
committed
Reorganize benchmark graphs by language and add 10-client concurrency level
- Separate graphs by language (Java/Ruby) to avoid apples-to-oranges comparisons - Add 10-client concurrency level (1, 10, 100) for better scaling visibility - README now shows RPS-only graphs (12 total) organized by Language → Concurrency - Move all latency percentiles (p50, p95, p99, p999) to dedicated detail pages - Add --language filter to generate_graphs.py for per-language graph generation - New graph directory structure: graphs/{language}/{concurrency}/ New files: configs/workloads/reference/basic-standalone-10-clients.json docs/BENCHMARKS_JAVA.md docs/BENCHMARKS_RUBY.md Signed-off-by: ikolomi <ikolomin@amazon.com>
1 parent 1f0e777 commit 33b6f48

File tree

7 files changed

+468
-49
lines changed

7 files changed

+468
-49
lines changed

‎.github/benchmark-config/workloads.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
[
22
"configs/workloads/reference/basic-standalone-single-client.json",
3+
"configs/workloads/reference/basic-standalone-10-clients.json",
34
"configs/workloads/reference/basic-standalone-100-clients.json"
45
]

‎.github/workflows/benchmark.yml‎

Lines changed: 52 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ jobs:
2222
- configs/drivers/reference/valkey-glide.json
2323
workload:
2424
- configs/workloads/reference/basic-standalone-single-client.json
25+
- configs/workloads/reference/basic-standalone-10-clients.json
2526
- configs/workloads/reference/basic-standalone-100-clients.json
2627

2728
steps:
@@ -96,6 +97,7 @@ jobs:
9697
- configs/drivers/reference/valkey-glide-ruby.json
9798
workload:
9899
- configs/workloads/reference/basic-standalone-single-client.json
100+
- configs/workloads/reference/basic-standalone-10-clients.json
99101
- configs/workloads/reference/basic-standalone-100-clients.json
100102

101103
steps:
@@ -185,22 +187,66 @@ jobs:
185187
echo "Downloaded results:"
186188
find results -name "*.ndjson" -type f
187189
188-
- name: Generate graphs - Single Client
190+
# Java graphs — per concurrency level
191+
- name: Generate Java graphs - 1 Client
189192
run: |
190193
python scripts/generate_graphs.py \
191194
--results "results/github-runner/reference/*-basic-standalone-single-client.ndjson" \
192-
--output graphs/single-client/ \
195+
--output graphs/java/1-client/ \
193196
--phase STEADY \
194-
--workload "Single Client" \
197+
--language java \
198+
--workload "Java - 1 Client" \
195199
--commit-id ${{ github.sha }}
196200
197-
- name: Generate graphs - 100 Clients
201+
- name: Generate Java graphs - 10 Clients
202+
run: |
203+
python scripts/generate_graphs.py \
204+
--results "results/github-runner/reference/*-basic-standalone-10-clients.ndjson" \
205+
--output graphs/java/10-clients/ \
206+
--phase STEADY \
207+
--language java \
208+
--workload "Java - 10 Clients" \
209+
--commit-id ${{ github.sha }}
210+
211+
- name: Generate Java graphs - 100 Clients
212+
run: |
213+
python scripts/generate_graphs.py \
214+
--results "results/github-runner/reference/*-basic-standalone-100-clients.ndjson" \
215+
--output graphs/java/100-clients/ \
216+
--phase STEADY \
217+
--language java \
218+
--workload "Java - 100 Clients" \
219+
--commit-id ${{ github.sha }}
220+
221+
# Ruby graphs — per concurrency level
222+
- name: Generate Ruby graphs - 1 Client
223+
run: |
224+
python scripts/generate_graphs.py \
225+
--results "results/github-runner/reference/*-basic-standalone-single-client.ndjson" \
226+
--output graphs/ruby/1-client/ \
227+
--phase STEADY \
228+
--language ruby \
229+
--workload "Ruby - 1 Client" \
230+
--commit-id ${{ github.sha }}
231+
232+
- name: Generate Ruby graphs - 10 Clients
233+
run: |
234+
python scripts/generate_graphs.py \
235+
--results "results/github-runner/reference/*-basic-standalone-10-clients.ndjson" \
236+
--output graphs/ruby/10-clients/ \
237+
--phase STEADY \
238+
--language ruby \
239+
--workload "Ruby - 10 Clients" \
240+
--commit-id ${{ github.sha }}
241+
242+
- name: Generate Ruby graphs - 100 Clients
198243
run: |
199244
python scripts/generate_graphs.py \
200245
--results "results/github-runner/reference/*-basic-standalone-100-clients.ndjson" \
201-
--output graphs/100-clients/ \
246+
--output graphs/ruby/100-clients/ \
202247
--phase STEADY \
203-
--workload "100 Clients" \
248+
--language ruby \
249+
--workload "Ruby - 100 Clients" \
204250
--commit-id ${{ github.sha }}
205251
206252
- name: Upload graphs

‎README.md‎

Lines changed: 57 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -4,41 +4,45 @@ A multi-language benchmark suite for RESP protocol (Redis/Valkey) compatible dat
44

55
## Performance Comparison
66

7-
The graphs below show performance comparisons of Java client libraries running on GitHub Actions runners. Results are automatically updated via CI.
7+
The graphs below show throughput (RPS) comparisons across client libraries, separated by language. Results are automatically updated via CI on GitHub Actions runners.
88

9-
### Single Client (1 connection)
9+
### Java Clients
1010

11-
| Throughput |
12-
|------------|
13-
| ![SET RPS - Single Client](graphs/single-client/rps-SET.png) |
14-
| ![GET RPS - Single Client](graphs/single-client/rps-GET.png) |
11+
#### Single Client (1 connection)
1512

16-
| Latency P50 |
17-
|-------------|
18-
| ![SET Latency P50 - Single Client](graphs/single-client/latency-p50-SET.png) |
19-
| ![GET Latency P50 - Single Client](graphs/single-client/latency-p50-GET.png) |
13+
![SET RPS - Java 1 Client](graphs/java/1-client/rps-SET.png)
14+
![GET RPS - Java 1 Client](graphs/java/1-client/rps-GET.png)
2015

21-
| Latency P99 |
22-
|-------------|
23-
| ![SET Latency P99 - Single Client](graphs/single-client/latency-p99-SET.png) |
24-
| ![GET Latency P99 - Single Client](graphs/single-client/latency-p99-GET.png) |
16+
#### 10 Concurrent Clients
2517

26-
### 100 Concurrent Clients
18+
![SET RPS - Java 10 Clients](graphs/java/10-clients/rps-SET.png)
19+
![GET RPS - Java 10 Clients](graphs/java/10-clients/rps-GET.png)
2720

28-
| Throughput |
29-
|------------|
30-
| ![SET RPS - 100 Clients](graphs/100-clients/rps-SET.png) |
31-
| ![GET RPS - 100 Clients](graphs/100-clients/rps-GET.png) |
21+
#### 100 Concurrent Clients
3222

33-
| Latency P50 |
34-
|-------------|
35-
| ![SET Latency P50 - 100 Clients](graphs/100-clients/latency-p50-SET.png) |
36-
| ![GET Latency P50 - 100 Clients](graphs/100-clients/latency-p50-GET.png) |
23+
![SET RPS - Java 100 Clients](graphs/java/100-clients/rps-SET.png)
24+
![GET RPS - Java 100 Clients](graphs/java/100-clients/rps-GET.png)
3725

38-
| Latency P99 |
39-
|-------------|
40-
| ![SET Latency P99 - 100 Clients](graphs/100-clients/latency-p99-SET.png) |
41-
| ![GET Latency P99 - 100 Clients](graphs/100-clients/latency-p99-GET.png) |
26+
> 📊 [Full Java latency breakdown (p50, p95, p99, p999)](docs/BENCHMARKS_JAVA.md)
27+
28+
### Ruby Clients
29+
30+
#### Single Client (1 connection)
31+
32+
![SET RPS - Ruby 1 Client](graphs/ruby/1-client/rps-SET.png)
33+
![GET RPS - Ruby 1 Client](graphs/ruby/1-client/rps-GET.png)
34+
35+
#### 10 Concurrent Clients
36+
37+
![SET RPS - Ruby 10 Clients](graphs/ruby/10-clients/rps-SET.png)
38+
![GET RPS - Ruby 10 Clients](graphs/ruby/10-clients/rps-GET.png)
39+
40+
#### 100 Concurrent Clients
41+
42+
![SET RPS - Ruby 100 Clients](graphs/ruby/100-clients/rps-SET.png)
43+
![GET RPS - Ruby 100 Clients](graphs/ruby/100-clients/rps-GET.png)
44+
45+
> 📊 [Full Ruby latency breakdown (p50, p95, p99, p999)](docs/BENCHMARKS_RUBY.md)
4246
4347
> **Note**: These benchmarks run on shared GitHub Actions runners. Results may have variance between runs due to noisy neighbor effects. The graphs show averages across multiple runs (n=count shown in labels).
4448
@@ -55,7 +59,7 @@ The graphs below show performance comparisons of Java client libraries running o
5559
| Language | Status | Supported Drivers |
5660
|----------|--------|-------------------|
5761
| Java | ✅ Ready | Jedis, Lettuce, Valkey-Glide, Redisson, Spring Data Valkey/Redis |
58-
| Ruby | ✅ Ready | redis-rb |
62+
| Ruby | ✅ Ready | redis-rb, valkey-glide-ruby |
5963
| Python | 🚧 Planned | redis-py, aioredis, valkey-glide |
6064
| Go | 📋 Future | go-redis, rueidis |
6165
| Node.js | 📋 Future | ioredis, node-redis |
@@ -124,7 +128,18 @@ resp-bench/
124128
├── docs/ # Documentation
125129
│ ├── ARCHITECTURE.md # System architecture
126130
│ ├── ADDING_LANGUAGE.md # Guide for adding new languages
127-
│ └── CONFIG_SPECIFICATION.md # Configuration format spec
131+
│ ├── CONFIG_SPECIFICATION.md # Configuration format spec
132+
│ ├── BENCHMARKS_JAVA.md # Full Java benchmark details (all percentiles)
133+
│ └── BENCHMARKS_RUBY.md # Full Ruby benchmark details (all percentiles)
134+
├── graphs/ # Auto-generated benchmark graphs
135+
│ ├── java/ # Java client graphs
136+
│ │ ├── 1-client/ # Single client results
137+
│ │ ├── 10-clients/ # 10 concurrent clients
138+
│ │ └── 100-clients/ # 100 concurrent clients
139+
│ └── ruby/ # Ruby client graphs
140+
│ ├── 1-client/
141+
│ ├── 10-clients/
142+
│ └── 100-clients/
128143
└── output/ # Default metrics output directory
129144
```
130145

@@ -239,14 +254,23 @@ Generate performance comparison graphs from benchmark results:
239254
# Install Python dependencies
240255
pip install matplotlib numpy
241256

242-
# Generate graphs from all results (auto-detects latest CI run)
257+
# Generate graphs for Java clients only
243258
python scripts/generate_graphs.py \
244259
--results results/github-runner/reference/*.ndjson \
245-
--output graphs/ \
260+
--output graphs/java/1-client/ \
261+
--phase STEADY \
262+
--language java \
263+
--workload "Java - 1 Client"
264+
265+
# Generate graphs for Ruby clients only
266+
python scripts/generate_graphs.py \
267+
--results results/github-runner/reference/*.ndjson \
268+
--output graphs/ruby/100-clients/ \
246269
--phase STEADY \
247-
--workload "My Benchmark"
270+
--language ruby \
271+
--workload "Ruby - 100 Clients"
248272

249-
# Generate graphs for a specific commit
273+
# Generate graphs for a specific commit (all languages)
250274
python scripts/generate_graphs.py \
251275
--results results/github-runner/reference/*.ndjson \
252276
--output graphs/ \

‎configs/workloads/reference/basic-standalone-10-clients.json‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
{
2+
"schema_version": "1.0",
3+
"benchmark_profile": {
4+
"name": "Basic Standalone - 10 Clients",
5+
"description": "Reference benchmark: 1M requests, 1M keys, 512B data, 10 connections",
6+
"version": "1.0.0"
7+
},
8+
"phases": [
9+
{
10+
"id": "WARMUP",
11+
"description": "Populate keys with sequential SET operations",
12+
"connections": 10,
13+
"cps_limit": -1,
14+
"rps_limit": -1,
15+
"completion": {
16+
"type": "requests",
17+
"requests": 100000
18+
},
19+
"keyspace": {
20+
"keys_count": 1000000,
21+
"key_size_bytes": 16,
22+
"key_prefix": "bench:",
23+
"generation_alg": "sequential_int"
24+
},
25+
"commands": [
26+
{
27+
"command": "set",
28+
"weight": 1.0,
29+
"data_size_bytes": 512
30+
}
31+
]
32+
},
33+
{
34+
"id": "STEADY",
35+
"description": "Steady-state workload with 50% GET, 50% SET",
36+
"connections": 10,
37+
"cps_limit": -1,
38+
"rps_limit": -1,
39+
"completion": {
40+
"type": "requests",
41+
"requests": 1000000
42+
},
43+
"keyspace": {
44+
"keys_count": 1000000,
45+
"key_size_bytes": 16,
46+
"key_prefix": "bench:",
47+
"generation_alg": "uniform_rand",
48+
"seed": 12345
49+
},
50+
"commands": [
51+
{
52+
"command": "get",
53+
"weight": 0.5
54+
},
55+
{
56+
"command": "set",
57+
"weight": 0.5,
58+
"data_size_bytes": 512
59+
}
60+
]
61+
}
62+
]
63+
}

0 commit comments

Comments
 (0)