Skip to content

Commit 14873d6

Browse files
committed
docs(readme): verified before/after benchmarks for all 5 opt-in wrappers
Measured 2026-04-22, N=3 per data-point, fresh state dir per cold column, against fixtures where sensible and real artifacts otherwise. No borrowed numbers, no estimates. Results (avg of 3 runs): wrapper bare cold warm warm vs bare sbt-direct oneshot 1661ms 4922ms 3600ms SLOWER sbt-direct bsp 1661ms 1285ms 131ms ~13× faster dotnet-direct 1172ms 1739ms 555ms ~2× faster prettier-direct 211ms 1278ms 95ms ~2× faster eslint-direct 274ms 1252ms 88ms ~3× faster scalafmt-direct 86ms 1243ms 112ms ~same Honest interpretation included: - sbt oneshot is SLOWER than bare sbt (sbt's own launcher daemon caches classpath; our oneshot spawns fresh JVM each call). Documented as "use bsp mode whenever possible". - dotnet-direct is within 2× of bare because MSBuild build-server persists across bare invocations too. Shipped for uniform CLI contract + calls.log observability, not speed. - scalafmt-direct is ~same as bare (native binary is already fast); ship for consistency, not speed. - sbt bsp warm is the headline win: 131ms vs bare's 207ms-4.5s variable range. - prettier + eslint warm are 2-3× wins because the daemon keeps require()d packages hot in memory. sbt bsp "cold" caveat called out explicitly: 1285ms is first /call after fresh state dir WITH a still-warm sbt JVM from prior session. Genuine from-scratch cold (Ivy/Coursier resolve + JVM boot + BSP init) is 15-90s on a fresh checkout. All subsequent calls land in the warm band. Bench script lived in /tmp/claude-501/bench.sh; not committed (it's machine-specific timing infrastructure, not part of the repo's test surface).
1 parent 461a8b2 commit 14873d6

1 file changed

Lines changed: 56 additions & 0 deletions

File tree

README.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,62 @@ The point isn't the specific numbers — it's the order-of-magnitude
7575
gap between a persistent HTTP proxy and the minimum cost of a
7676
per-call tool turn.
7777

78+
### Opt-in wrappers (sbt, dotnet, prettier, eslint, scalafmt)
79+
80+
Measured 2026-04-22 on macOS 26.4.1 arm64, N=3 iterations each. Cold
81+
= fresh coordinator (state dir wiped); warm = coordinator already
82+
running. "Bare" = invoking the underlying tool directly from a shell
83+
(whatever caching that tool's launcher does is already included).
84+
85+
| wrapper | bare tool avg | direct cold avg | direct warm avg | warm vs bare |
86+
|---|---|---|---|---|
87+
| `sbt-direct` oneshot | 1661ms ¹ | 4922ms | 3600ms | **slower** — see note |
88+
| `sbt-direct` bsp | 1661ms ¹ | 1285ms ² | **131ms** | **~13× faster** |
89+
| `dotnet-direct` | 1172ms ³ | 1739ms | 555ms | **~2× faster** |
90+
| `prettier-direct` | 211ms | 1278ms | 95ms | **~2× faster** |
91+
| `eslint-direct` | 274ms | 1252ms | 88ms | **~3× faster** |
92+
| `scalafmt-direct` | 86ms | 1243ms | 112ms | **~same** (native already fast) |
93+
94+
¹ sbt's own launcher daemon caches classpath across invocations, so
95+
"bare sbt" numbers range from 207ms (daemon hot) to 4.5s (daemon
96+
cold). 1661ms is the 3-run average.
97+
98+
² `bsp cold` here measures only the first `/call` after a fresh state
99+
dir. The underlying sbt JVM was still warm from a prior session on
100+
this machine. **Genuine from-scratch cold** (new Ivy/Coursier
101+
resolve + JVM boot + BSP init) is 15-90s on a fresh checkout — once,
102+
per workspace. All subsequent calls land in the 131ms warm band.
103+
104+
³ MSBuild's build-server persists across invocations automatically, so
105+
"bare dotnet" is already warm on the 2nd+ call. 1172ms is the average
106+
of one cold + two warm runs.
107+
108+
### Where direct wrappers actually help
109+
110+
- **sbt bsp warm path** — 131ms vs bare sbt's unpredictable
111+
207ms-4.5s range. Biggest win by far.
112+
- **prettier + eslint warm** — 2-3× faster than bare because the
113+
daemon keeps `require('prettier')` / `new ESLint(...)` cached in
114+
memory.
115+
- **LSP wrappers** (py/ts/cs/java/vue) — covered by the LSP table
116+
above; the argument is against `LSP()` tool-harness round-trips,
117+
not against bare language-server invocation.
118+
119+
### Where direct wrappers don't help (or hurt)
120+
121+
- **sbt oneshot mode** — no persistence, so it strictly adds
122+
coordinator overhead on top of bare sbt. Exists only as a fallback
123+
for when BSP isn't configured in the workspace. **Use `bsp` mode
124+
(`SBT_DIRECT_MODE=bsp`) whenever possible.**
125+
- **dotnet-direct** — MSBuild's build-server gives bare dotnet the
126+
same warm-path benefit. Direct wrapper is within 2× of bare;
127+
shipping it is about uniform CLI contract + `calls.log` observability,
128+
not speed.
129+
- **scalafmt-direct** — scalafmt's native binary is already
130+
sub-200ms cold. Direct adds coordinator round-trip overhead that
131+
cancels the win. Ship it for consistency (and for access to
132+
scalafmt from the same harness as the other tools), not speed.
133+
78134
## Architecture
79135

80136
```

0 commit comments

Comments
 (0)