Skip to content

Commit 7b2ba00

Browse files
committed
docs(readme): refresh benchmark tables with 2026-06-25 v0.10.9 numbers
Re-ran the full 14-fixture cross-runtime bench on the standard 2C/4G cgroup (10000 req × 16 conc) on the same machine that produced the previous 2026-06-11 numbers. v0.10 series picked up wangshu CallInto / GlobalsSlot fast paths, outputPool (#119), and Redis cascade-safety (#137) — re-baseline so the README reflects measured reality. Changes worth calling out: - Three calibrated fixtures now listed instead of one. Until now the README collapsed the calibrated family to a single row, hiding the itemlua variant entirely. itemlua (3000 Lua calls/request) is the boundary-dominated workload that anchors the perf-evolution-roadmap "calibration fact 2 — end-to-end dilution" finding; it deserves to show up. - C++ headline lift: 1.8x → 1.9x against Go/Java on calibrated. P50 60.8ms vs 117/122ms is the more legible framing than the QPS ratio. - Synthetic small/medium movements are all within ±10 % run-to-run noise; the relative shape (Go highest at small, Java reverses on large_1000+) is unchanged. - Reproduce command now lists `make bench-cross-runtime` first. Source data: bench-results/report-20260625-090834.txt
1 parent 1fc1884 commit 7b2ba00

2 files changed

Lines changed: 51 additions & 39 deletions

File tree

README-en.md

Lines changed: 25 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -333,39 +333,45 @@ See `scripts/cross-validate.sh` for a complete production implementation.
333333

334334
## Benchmark
335335

336-
Cross-engine performance comparison (HTTP server mode, `scripts/bench-cross-runtime.sh`, 10000 requests × 16 concurrency, server cgroup-isolated to 2C/4G). `realistic_calibrated` is a production proxy fixture calibrated against real traffic; the rest are synthetic stress tests.
336+
Cross-engine performance comparison (HTTP server mode, `scripts/bench-cross-runtime.sh`, 10000 requests × 16 concurrency, server cgroup-isolated to 2C/4G, re-measured 2026-06-25 / v0.10.9). `realistic_*_calibrated*` fixtures are production-proxy benchmarks calibrated against real traffic; the rest are synthetic stress tests.
337337

338338
### Throughput (QPS)
339339

340340
| Fixture | Go | Java | C++ |
341341
|---|---|---|---|
342-
| small_010 (10 items) | 37078 | 5825 | 20794 |
343-
| small_050 (50 items) | 26976 | 5201 | 17244 |
344-
| small_100 (100 items) | 19585 | 4748 | 13904 |
345-
| medium_0100 (100 items) | 12025 | 3681 | 8578 |
346-
| medium_0500 (500 items) | 2921 | 2034 | 2938 |
347-
| medium_1000 (1000 items) | 1446 | 1360 | 1647 |
348-
| large_0100 (100 items) | 6395 | 2855 | 4855 |
349-
| large_0500 (500 items) | 1439 | 1439 | 1671 |
350-
| large_1000 (1000 items) | 728 | 917 | 902 |
351-
| large_5000 (5000 items) | 142 | 212 | 174 |
352-
| **realistic_calibrated (production proxy)** | **120** | **124** | **221** |
342+
| small_010 (10 items) | 36298 | 6318 | 20756 |
343+
| small_050 (50 items) | 27270 | 5336 | 17227 |
344+
| small_100 (100 items) | 19658 | 4607 | 13812 |
345+
| medium_0100 (100 items) | 12514 | 3589 | 8542 |
346+
| medium_0500 (500 items) | 3026 | 1965 | 2941 |
347+
| medium_1000 (1000 items) | 1513 | 1295 | 1656 |
348+
| large_0100 (100 items) | 7243 | 3064 | 5120 |
349+
| large_0500 (500 items) | 1684 | 1508 | 1773 |
350+
| large_1000 (1000 items) | 825 | 966 | 951 |
351+
| large_5000 (5000 items) | 155 | 213 | 175 |
352+
| realistic_for_you | 483 | 303 | 349 |
353+
| realistic_for_you_latency | 250 | 141 | 212 |
354+
| **realistic_for_you_calibrated (production proxy)** | **121** | **127** | **237** |
355+
| **realistic_for_you_calibrated_2c4g** | **121** | **124** | **224** |
356+
| **realistic_for_you_calibrated_itemlua** | **127** | **126** | **233** |
353357

354358
### P50 Latency (ms)
355359

356360
| Fixture | Go | Java | C++ |
357361
|---|---|---|---|
358-
| small_010 | 0.3 | 2.0 | 0.6 |
359-
| medium_0500 | 5.0 | 6.3 | 5.2 |
360-
| large_1000 | 20.5 | 14.8 | 16.1 |
361-
| large_5000 | 102.2 | 67.9 | 83.9 |
362-
| **realistic_calibrated** | **123.6** | **121.9** | **65.0** |
362+
| small_010 | 0.4 | 1.5 | 0.6 |
363+
| medium_0500 | 4.9 | 6.8 | 5.3 |
364+
| large_1000 | 18.2 | 14.3 | 15.3 |
365+
| large_5000 | 94.3 | 68.6 | 83.4 |
366+
| **realistic_for_you_calibrated** | **122.3** | **117.7** | **60.8** |
367+
| **realistic_for_you_calibrated_itemlua** | **117.1** | **119.5** | **61.5** |
363368

364369
Highlights:
365370

366-
- **C++ leads by ~1.8x on the production-calibrated scenario** (QPS 221 vs 120/124; P50 65ms vs ~122ms) — this is what the "benchmark runtime" positioning means
371+
- **C++ leads by ~1.9x on production-calibrated workloads** (calibrated QPS 237 vs 121/127; P50 60ms vs 117/122ms) — this is what the "benchmark runtime" positioning means
367372
- Go has the highest throughput on synthetic small/medium fixtures (lowest lightweight-request overhead); Java's JIT hot-loop optimization wins at large row counts (large_1000+)
368-
- Numbers evolve with versions. Reproduce with `scripts/bench-cross-runtime.sh --requests 10000 --concurrency 16`; reports land in `bench-results/`
373+
- itemlua (3000 Lua calls/request, boundary-dominated shape) is statistically flat against calibrated across all three engines — confirms the "per-item boundary dominates + end-to-end dilution" calibration fact (see `llmdoc/memory/decisions/perf-evolution-roadmap.md`)
374+
- Numbers evolve with versions. Reproduce with `make bench-cross-runtime` or `scripts/bench-cross-runtime.sh --requests 10000 --concurrency 16`; reports land in `bench-results/`
369375

370376
## Documentation
371377

README.md

Lines changed: 26 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -349,39 +349,45 @@ def normalize_json(text):
349349

350350
## Benchmark
351351

352-
跨引擎性能对比(HTTP server 模式,`scripts/bench-cross-runtime.sh`,10000 请求 × 16 并发,server 以 2C/4G cgroup 隔离)。`realistic_calibrated` 为按真实流量校准的生产 proxy fixture,其余为合成压测。
352+
跨引擎性能对比(HTTP server 模式,`scripts/bench-cross-runtime.sh`,10000 请求 × 16 并发,server 以 2C/4G cgroup 隔离,2026-06-25 / v0.10.9 复测)。`realistic_*_calibrated*` 系列为按真实流量校准的生产 proxy fixture,其余为合成压测。
353353

354354
### 吞吐量 (QPS)
355355

356356
| Fixture | Go | Java | C++ |
357357
|---|---|---|---|
358-
| small_010 (10 items) | 37078 | 5825 | 20794 |
359-
| small_050 (50 items) | 26976 | 5201 | 17244 |
360-
| small_100 (100 items) | 19585 | 4748 | 13904 |
361-
| medium_0100 (100 items) | 12025 | 3681 | 8578 |
362-
| medium_0500 (500 items) | 2921 | 2034 | 2938 |
363-
| medium_1000 (1000 items) | 1446 | 1360 | 1647 |
364-
| large_0100 (100 items) | 6395 | 2855 | 4855 |
365-
| large_0500 (500 items) | 1439 | 1439 | 1671 |
366-
| large_1000 (1000 items) | 728 | 917 | 902 |
367-
| large_5000 (5000 items) | 142 | 212 | 174 |
368-
| **realistic_calibrated (生产校准)** | **120** | **124** | **221** |
358+
| small_010 (10 items) | 36298 | 6318 | 20756 |
359+
| small_050 (50 items) | 27270 | 5336 | 17227 |
360+
| small_100 (100 items) | 19658 | 4607 | 13812 |
361+
| medium_0100 (100 items) | 12514 | 3589 | 8542 |
362+
| medium_0500 (500 items) | 3026 | 1965 | 2941 |
363+
| medium_1000 (1000 items) | 1513 | 1295 | 1656 |
364+
| large_0100 (100 items) | 7243 | 3064 | 5120 |
365+
| large_0500 (500 items) | 1684 | 1508 | 1773 |
366+
| large_1000 (1000 items) | 825 | 966 | 951 |
367+
| large_5000 (5000 items) | 155 | 213 | 175 |
368+
| realistic_for_you | 483 | 303 | 349 |
369+
| realistic_for_you_latency | 250 | 141 | 212 |
370+
| **realistic_for_you_calibrated (生产校准)** | **121** | **127** | **237** |
371+
| **realistic_for_you_calibrated_2c4g** | **121** | **124** | **224** |
372+
| **realistic_for_you_calibrated_itemlua** | **127** | **126** | **233** |
369373

370374
### P50 延迟 (ms)
371375

372376
| Fixture | Go | Java | C++ |
373377
|---|---|---|---|
374-
| small_010 | 0.3 | 2.0 | 0.6 |
375-
| medium_0500 | 5.0 | 6.3 | 5.2 |
376-
| large_1000 | 20.5 | 14.8 | 16.1 |
377-
| large_5000 | 102.2 | 67.9 | 83.9 |
378-
| **realistic_calibrated** | **123.6** | **121.9** | **65.0** |
378+
| small_010 | 0.4 | 1.5 | 0.6 |
379+
| medium_0500 | 4.9 | 6.8 | 5.3 |
380+
| large_1000 | 18.2 | 14.3 | 15.3 |
381+
| large_5000 | 94.3 | 68.6 | 83.4 |
382+
| **realistic_for_you_calibrated** | **122.3** | **117.7** | **60.8** |
383+
| **realistic_for_you_calibrated_itemlua** | **117.1** | **119.5** | **61.5** |
379384

380385
要点:
381386

382-
- **生产校准场景下 C++ 领先约 1.8x**(QPS 221 vs 120/124;P50 65ms vs ~122ms),这是"标杆运行时"定位的体现
383-
- 合成 small/medium 场景 Go 吞吐最高(轻量请求路径开销最低);大行数场景(large_1000+)Java 的 JIT 热循环优化使其反超
384-
- 各引擎数字会随版本演进,复现方式:`scripts/bench-cross-runtime.sh --requests 10000 --concurrency 16`,报告落在 `bench-results/`
387+
- **生产校准场景下 C++ 领先约 1.9x**(calibrated QPS 237 vs 121/127;P50 60ms vs 117/122ms),这是"标杆运行时"定位的体现
388+
- 合成 small/medium 场景 Go 吞吐最高(轻量请求路径开销最低);大行数场景(large_1000+)Java 的 JIT 热循环优化反超
389+
- itemlua(3000 调用/请求的 boundary-dominated 形状)与 calibrated 在三引擎都统计持平,符合"per-item 边界主导 + 端到端稀释"的校准事实(详见 `llmdoc/memory/decisions/perf-evolution-roadmap.md`
390+
- 各引擎数字会随版本演进,复现方式:`make bench-cross-runtime``scripts/bench-cross-runtime.sh --requests 10000 --concurrency 16`,报告落在 `bench-results/`
385391

386392
## 文档
387393

0 commit comments

Comments
 (0)