1- # B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS)
1+ # B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS, gin/Go )
22
33Measures whether the CALM-scripted refactor workflow (` edit_context ` →
44rename at each real reference → ` diff_impact ` ) completes a real rename task
@@ -27,16 +27,18 @@ directly:
2727| python | flask (pallets/flask) | ` uv sync --frozen ` | ` uv run pytest -q ` |
2828| javascript | express (expressjs/express) | ` npm install ` | ` npm test ` |
2929| typescript | zod (colinhacks/zod) | ` pnpm install ` | ` pnpm test ` |
30+ | go | gin (gin-gonic/gin) | ` go build ./... ` | ` go test ./... ` |
3031
31- ** Go/gin was skipped this round** — ` go ` is not installed in this environment
32- and passwordless ` sudo ` is unavailable to install it (verified live:
33- ` apt-cache policy golang-go ` shows a candidate package, but ` sudo -n true `
34- fails). A real, honestly-reported environment gap, not a benchmark bug —
35- gin is the documented next addition once a Go toolchain is available.
32+ ** Go/gin was initially skipped** in this session's first Phase-2 pass — ` go `
33+ wasn't installed and passwordless ` sudo ` was unavailable (verified live:
34+ ` apt-cache policy golang-go ` showed a candidate package, but ` sudo -n true `
35+ failed). The user then provided sudo access specifically to unblock this
36+ (one-time, not persisted anywhere), Go 1.22.2 was installed via
37+ ` apt-get install golang-go ` , and gin was added the same day.
3638
3739** Verified live, not assumed — every build/test command below was confirmed
38- correct by actually running it, not by reading a package.json/pyproject.toml
39- and guessing:**
40+ correct by actually running it, not by reading a package.json/pyproject.toml/
41+ go.mod and guessing:**
4042- flask needs ` uv sync --frozen ` + ` uv run pytest ` , ** not** bare
4143 ` pip install pytest ` — a first attempt with bare pip grabbed the latest
4244 pytest, whose internal ` _pytest.monkeypatch.notset ` API (removed upstream)
@@ -47,6 +49,10 @@ and guessing:**
4749- express has no committed lockfile — plain ` npm install ` is correct.
4850- zod uses ` pnpm ` (` pnpm-lock.yaml ` present, not ` package-lock.json ` ) — a
4951 bare ` npm install ` would not respect its lockfile.
52+ - gin's ` go.mod ` requires Go 1.25.0, newer than the installed 1.22.2 —
53+ ` GOTOOLCHAIN=auto ` (Go's default since 1.21) transparently downloaded
54+ 1.25.0 on first ` go build ` , no manual intervention needed. Verified by
55+ actually running the build, not assumed from the version mismatch alone.
5056
5157## Isolation
5258
@@ -88,24 +94,29 @@ building this, in two separate iterations:
8894 by widening the pattern to match the bare identifier regardless of what
8995 follows it.
9096
91- ## Results (all 4 tasks, current methodology)
97+ ## Results (all 5 tasks, current methodology)
9298
9399| task | baseline | naive build_pass | naive recall | naive tool_calls | calm build_pass | calm recall | calm tool_calls |
94100| ---| ---| ---| ---| ---| ---| ---| ---|
95101| rename_fd_pattern_matches_leading_dot | green | True | 1.0 | 3 | True | 1.0 | 4 |
96102| rename_flask_from_prefixed_env | green | True | 1.0 | 4 | True | 1.0 | 5 |
103+ | rename_gin_clean_path | green | True | 1.0 | 3 | True | 1.0 | 4 |
97104| rename_express_set_charset | green | True | 1.0 | 4 | ** False** | ** 0.667** | 4 |
98105| rename_zod_prettify_error | green | True | 1.0 | 5 | ** False** | ** 0.5** | 4 |
99106
100- ### Phase 1 (fd, flask): an honest tie
107+ ### Phase 1 + gin (fd, flask, gin ): an honest tie
101108
102- Both arms hit perfect recall + a passing build on both tasks, and naive even
103- uses fewer tool calls. Reported as measured, not hidden (project policy — cf.
104- B6's ` find_callers ` =0% precedent): both Phase-1 symbols are distinctive
105- enough (unique corpus-wide, picked via B12's ` sample_distinctive ` filter)
106- that a repo-wide ` git grep ` already finds every reference. This doesn't mean
107- CALM has no advantage here — it means these two symbols don't exercise the
108- case where an advantage would show up.
109+ All three arms hit perfect recall + a passing build, and naive uses fewer or
110+ equal tool calls. Reported as measured, not hidden (project policy — cf.
111+ B6's ` find_callers ` =0% precedent): all three symbols are distinctive enough
112+ (unique corpus-wide, picked via B12's ` sample_distinctive ` filter) that a
113+ repo-wide ` git grep ` already finds every reference. gin's case additionally
114+ confirms * why* : Go has no equivalent of JS's ` require('./mod').fn(...) `
115+ property-access call shape for same-package calls (Go's intra-package calls
116+ are always bare identifiers), so there's no structural opportunity for the
117+ property-access gap seen in JS to appear here at all. This doesn't mean CALM
118+ has no advantage on these three — it means these symbols don't exercise the
119+ case where an advantage (or the JS/TS gap) would show up.
109120
110121### Phase 2 (express, zod): a real, reproducible finding — ` edit_context ` 's
111122` callers() ` alone is not sufficient for a complete rename
@@ -204,21 +215,20 @@ benchmarks/.venv/bin/python benchmarks/b7_task_correctness/run_benchmark.py --ta
204215```
205216
206217Preconditions: network access (fresh ` git clone --local ` of the pinned
207- sources, plus ` cargo ` 's crates.io fetch, ` uv sync ` 's and ` npm ` /`pnpm
208- install ` 's package resolution on first run); ` uv` on PATH for the flask task,
209- ` pnpm ` on PATH for the zod task. Go/gin is not runnable until a Go toolchain
210- is installed (see Corpora above) .
218+ sources, plus ` cargo ` 's crates.io fetch, ` uv sync ` 's, ` npm ` /` pnpm install ` 's,
219+ and ` go build ` 's module resolution on first run); ` uv ` on PATH for the flask
220+ task, ` pnpm ` on PATH for the zod task, a Go toolchain ( ` GOTOOLCHAIN=auto `
221+ handles a version mismatch against ` go.mod ` ) for the gin task .
211222
212223` benchmarks/b7_task_correctness/.work ` (B12's own convention) is ** not** used
213224here — see Isolation above; work copies land in ` ../calm-b7-work/ ` instead,
214225which is not committed and safe to delete between runs.
215226
216227## Next steps
217228
218- 1 . ** gin/Go** once a Go toolchain is available (Phase 2 completion).
219- 2 . ** spring-petclinic/Java** (Phase 3 — Maven+JVM, expected to be the
229+ 1 . ** spring-petclinic/Java** (Phase 3 — Maven+JVM, expected to be the
220230 heaviest/flakiest setup of the five, per the design spec).
221- 3 . Investigate the express ` setCharset ` call-graph gap directly in
231+ 2 . Investigate the express ` setCharset ` call-graph gap directly in
222232 ` parser.rs ` 's JS/TS call-site extraction (property-access call through a
223233 required module's bare identifier vs. a destructured bare-name call to
224234 the same export) — a candidate root-cause worth its own session, not
0 commit comments