|
1 | 1 | # Performance report: the serving regime, measured |
2 | 2 |
|
3 | | -**Scope.** This report collects the serving-path performance measurements of the SQL specializer (`DuckDBInferFn`) as of 2026-07-28: where a call's time actually goes, how the engine compares per call with DuckDB itself, what the Arrow boundary bought, and why the columnar execution core was built, measured, and then deliberately closed unmerged. Every number is a p50 from a release build on this project's benchmark harness (`benchmarks/bench_serving.py`, `scripts/bench_scaling.py` on branch `task-61-columnar-core`), taken after a three-way parity gate (specializer == DuckDB == handcrafted Python) confirmed the outputs agree — a scenario that disagrees aborts the bench before any timing. The headline, stated plainly: for calls of 1–1,000 rows the specializer is one to three orders of magnitude faster per call than invoking DuckDB, the crossover sits at a few thousand rows per call, and above that DuckDB wins by 3–5× — a boundary we have measured and chosen not to contest. |
| 3 | +**Scope.** This report collects the serving-path performance measurements of Confit (`DuckDBInferFn`) as of 2026-07-28: where a call's time actually goes, how the engine compares per call with DuckDB itself, what the Arrow boundary bought, and why the columnar execution core was built, measured, and then deliberately closed unmerged. Every number is a p50 from a release build on this project's benchmark harness (`benchmarks/bench_serving.py`, `scripts/bench_scaling.py` on branch `task-61-columnar-core`), taken after a three-way parity gate (Confit == DuckDB == handcrafted Python) confirmed the outputs agree — a scenario that disagrees aborts the bench before any timing. The headline, stated plainly: for calls of 1–1,000 rows Confit is one to three orders of magnitude faster per call than invoking DuckDB, the crossover sits at a few thousand rows per call, and above that DuckDB wins by 3–5× — a boundary we have measured and chosen not to contest. |
4 | 4 |
|
5 | 5 | ## 1. The serving problem |
6 | 6 |
|
7 | | -The product is per-call latency at small batches. An inference service receives one request — one row, or a few dozen, occasionally a thousand — and must answer now. Throughput over a quarter-million-row table is an analytics problem; DuckDB already solves it well. The specializer exists for the other regime: `prepare(sql, static_tables) -> f` once at fit time (seconds are fine), then `f(batch)` millions of times with a nanosecond budget (`docs/superpowers/specs/2026-07-25-sql-specializer-design.md`). |
| 7 | +The product is per-call latency at small batches. An inference service receives one request — one row, or a few dozen, occasionally a thousand — and must answer now. Throughput over a quarter-million-row table is an analytics problem; DuckDB already solves it well. Confit exists for the other regime: `prepare(sql, static_tables) -> f` once at fit time (seconds are fine), then `f(batch)` millions of times with a nanosecond budget (`docs/superpowers/specs/2026-07-25-sql-specializer-design.md`). |
8 | 8 |
|
9 | 9 | The performance numbers below are only meaningful against the correctness backdrop, so it is stated once: the engine's contract is *serve bit-exact with DuckDB 1.5.5 or refuse loudly at build time* (`docs/known-limitations.md`). The mined-corpus replay (`packages/confit/tests/test_corpus_replay.py`) enforces a three-outcome contract — match, clean-unsupported, or FAIL — with zero FAILs required. The match ladder over the waves ran 53 → 395 → 505 → 511 → 529 → 546 → 550 of 678 statements, zero FAILs throughout (wave census anchors in `docs/superpowers/specs/2026-07-26-wave5-structural-pins.md`, `2026-07-27-waveB-regexp-pins.md`, `2026-07-28-waveA-structural-tails.md`, `2026-07-28-stageB-multiplicity-pins.md`; current 550/678 recorded in `docs/known-limitations.md`). Nothing below was bought by relaxing that. |
10 | 10 |
|
@@ -80,7 +80,7 @@ And house_prices, the scenario where the columnar lane does best: |
80 | 80 | | 16,384 | 4,379 | 5,792 | 5,592 | 1,815 | |
81 | 81 | | 262,144 | 4,720 | 5,987 | 6,505 | 1,163 | |
82 | 82 |
|
83 | | -Readings, all visible in the raw JSON. Our per-row cost is essentially flat from n=1,024 upward (~3–5 µs/row depending on scenario) — the specializer does not get faster per row with scale, because it was never built to. DuckDB's per-row cost falls three orders of magnitude as its fixed cost amortises and its parallelism engages, crossing us at ~4k rows/call on titanic, fraud_txn and store_sales, and between 4k and 16k on house_prices (per call at 4,096: house 14.58 ms us vs 15.58 ms DuckDB — still ours; at 16,384: 71.8 ms vs 29.7 ms — DuckDB's). At the top end DuckDB wins by ~5× (titanic n=262,144: 700 vs 3,847 ns/row). The Arrow lane's fixed cost is stark at n=1 (189–445 µs across scenarios against a 4–10 µs row-path call) and fully amortised by ~1k, where it beats the twin on house_prices and titanic and trails it modestly on the string-heavy scenarios. One run-to-run honesty note: house_prices n=1,024 measured 2.80 ms per call in the TASK-60 bench and 3.79 ms in this sweep — different runs on different days with the columnar core active in the latter; treat gaps of that size between editions as within run variance, not as findings. |
| 83 | +Readings, all visible in the raw JSON. Our per-row cost is essentially flat from n=1,024 upward (~3–5 µs/row depending on scenario) — Confit does not get faster per row with scale, because it was never built to. DuckDB's per-row cost falls three orders of magnitude as its fixed cost amortises and its parallelism engages, crossing us at ~4k rows/call on titanic, fraud_txn and store_sales, and between 4k and 16k on house_prices (per call at 4,096: house 14.58 ms us vs 15.58 ms DuckDB — still ours; at 16,384: 71.8 ms vs 29.7 ms — DuckDB's). At the top end DuckDB wins by ~5× (titanic n=262,144: 700 vs 3,847 ns/row). The Arrow lane's fixed cost is stark at n=1 (189–445 µs across scenarios against a 4–10 µs row-path call) and fully amortised by ~1k, where it beats the twin on house_prices and titanic and trails it modestly on the string-heavy scenarios. One run-to-run honesty note: house_prices n=1,024 measured 2.80 ms per call in the TASK-60 bench and 3.79 ms in this sweep — different runs on different days with the columnar core active in the latter; treat gaps of that size between editions as within run variance, not as findings. |
84 | 84 |
|
85 | 85 | ## 7. Methodology |
86 | 86 |
|
|
0 commit comments