Skip to content

Commit 555126a

Browse files
Your Nameclaude
andcommitted
feat(benchmarks): B7 Phase 2 complete -- add gin/Go task
Go toolchain wasn't available earlier this session (no `go`, no passwordless sudo); the user then provided sudo access specifically to unblock this (one-time, not persisted anywhere), Go 1.22.2 was installed via `apt-get install golang-go`, and gin (gin-gonic/gin) was added the same day completing Phase 2's 5-language set (Rust/Python/JS/TS/Go). gin's go.mod requires Go 1.25.0, newer than the installed 1.22.2 -- GOTOOLCHAIN=auto (Go's default since 1.21) transparently downloaded 1.25.0 on first `go build`, verified live rather than assumed from the version mismatch alone. rename_gin_clean_path (cleanPath, path.go -> gin.go + path_test.go) ties at 100% recall + passing build on both arms -- consistent with, and explained by, the express/zod finding from the prior commit: Go has no equivalent of JS's require('./mod').fn(...) property-access call shape for same-package calls (intra-package calls are always bare identifiers), so there's no structural opening for the property-access call-graph gap seen in JS to appear here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 76b385e commit 555126a

3 files changed

Lines changed: 66 additions & 31 deletions

File tree

benchmarks/b7_task_correctness/README.md

Lines changed: 34 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS)
1+
# B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS, gin/Go)
22

33
Measures whether the CALM-scripted refactor workflow (`edit_context`
44
rename at each real reference → `diff_impact`) completes a real rename task
@@ -27,16 +27,18 @@ directly:
2727
| python | flask (pallets/flask) | `uv sync --frozen` | `uv run pytest -q` |
2828
| javascript | express (expressjs/express) | `npm install` | `npm test` |
2929
| typescript | zod (colinhacks/zod) | `pnpm install` | `pnpm test` |
30+
| go | gin (gin-gonic/gin) | `go build ./...` | `go test ./...` |
3031

31-
**Go/gin was skipped this round**`go` is not installed in this environment
32-
and passwordless `sudo` is unavailable to install it (verified live:
33-
`apt-cache policy golang-go` shows a candidate package, but `sudo -n true`
34-
fails). A real, honestly-reported environment gap, not a benchmark bug —
35-
gin is the documented next addition once a Go toolchain is available.
32+
**Go/gin was initially skipped** in this session's first Phase-2 pass — `go`
33+
wasn't installed and passwordless `sudo` was unavailable (verified live:
34+
`apt-cache policy golang-go` showed a candidate package, but `sudo -n true`
35+
failed). The user then provided sudo access specifically to unblock this
36+
(one-time, not persisted anywhere), Go 1.22.2 was installed via
37+
`apt-get install golang-go`, and gin was added the same day.
3638

3739
**Verified live, not assumed — every build/test command below was confirmed
38-
correct by actually running it, not by reading a package.json/pyproject.toml
39-
and guessing:**
40+
correct by actually running it, not by reading a package.json/pyproject.toml/
41+
go.mod and guessing:**
4042
- flask needs `uv sync --frozen` + `uv run pytest`, **not** bare
4143
`pip install pytest` — a first attempt with bare pip grabbed the latest
4244
pytest, whose internal `_pytest.monkeypatch.notset` API (removed upstream)
@@ -47,6 +49,10 @@ and guessing:**
4749
- express has no committed lockfile — plain `npm install` is correct.
4850
- zod uses `pnpm` (`pnpm-lock.yaml` present, not `package-lock.json`) — a
4951
bare `npm install` would not respect its lockfile.
52+
- gin's `go.mod` requires Go 1.25.0, newer than the installed 1.22.2 —
53+
`GOTOOLCHAIN=auto` (Go's default since 1.21) transparently downloaded
54+
1.25.0 on first `go build`, no manual intervention needed. Verified by
55+
actually running the build, not assumed from the version mismatch alone.
5056

5157
## Isolation
5258

@@ -88,24 +94,29 @@ building this, in two separate iterations:
8894
by widening the pattern to match the bare identifier regardless of what
8995
follows it.
9096

91-
## Results (all 4 tasks, current methodology)
97+
## Results (all 5 tasks, current methodology)
9298

9399
| task | baseline | naive build_pass | naive recall | naive tool_calls | calm build_pass | calm recall | calm tool_calls |
94100
|---|---|---|---|---|---|---|---|
95101
| rename_fd_pattern_matches_leading_dot | green | True | 1.0 | 3 | True | 1.0 | 4 |
96102
| rename_flask_from_prefixed_env | green | True | 1.0 | 4 | True | 1.0 | 5 |
103+
| rename_gin_clean_path | green | True | 1.0 | 3 | True | 1.0 | 4 |
97104
| rename_express_set_charset | green | True | 1.0 | 4 | **False** | **0.667** | 4 |
98105
| rename_zod_prettify_error | green | True | 1.0 | 5 | **False** | **0.5** | 4 |
99106

100-
### Phase 1 (fd, flask): an honest tie
107+
### Phase 1 + gin (fd, flask, gin): an honest tie
101108

102-
Both arms hit perfect recall + a passing build on both tasks, and naive even
103-
uses fewer tool calls. Reported as measured, not hidden (project policy — cf.
104-
B6's `find_callers`=0% precedent): both Phase-1 symbols are distinctive
105-
enough (unique corpus-wide, picked via B12's `sample_distinctive` filter)
106-
that a repo-wide `git grep` already finds every reference. This doesn't mean
107-
CALM has no advantage here — it means these two symbols don't exercise the
108-
case where an advantage would show up.
109+
All three arms hit perfect recall + a passing build, and naive uses fewer or
110+
equal tool calls. Reported as measured, not hidden (project policy — cf.
111+
B6's `find_callers`=0% precedent): all three symbols are distinctive enough
112+
(unique corpus-wide, picked via B12's `sample_distinctive` filter) that a
113+
repo-wide `git grep` already finds every reference. gin's case additionally
114+
confirms *why*: Go has no equivalent of JS's `require('./mod').fn(...)`
115+
property-access call shape for same-package calls (Go's intra-package calls
116+
are always bare identifiers), so there's no structural opportunity for the
117+
property-access gap seen in JS to appear here at all. This doesn't mean CALM
118+
has no advantage on these three — it means these symbols don't exercise the
119+
case where an advantage (or the JS/TS gap) would show up.
109120

110121
### Phase 2 (express, zod): a real, reproducible finding — `edit_context`'s
111122
`callers()` alone is not sufficient for a complete rename
@@ -204,21 +215,20 @@ benchmarks/.venv/bin/python benchmarks/b7_task_correctness/run_benchmark.py --ta
204215
```
205216

206217
Preconditions: network access (fresh `git clone --local` of the pinned
207-
sources, plus `cargo`'s crates.io fetch, `uv sync`'s and `npm`/`pnpm
208-
install`'s package resolution on first run); `uv` on PATH for the flask task,
209-
`pnpm` on PATH for the zod task. Go/gin is not runnable until a Go toolchain
210-
is installed (see Corpora above).
218+
sources, plus `cargo`'s crates.io fetch, `uv sync`'s, `npm`/`pnpm install`'s,
219+
and `go build`'s module resolution on first run); `uv` on PATH for the flask
220+
task, `pnpm` on PATH for the zod task, a Go toolchain (`GOTOOLCHAIN=auto`
221+
handles a version mismatch against `go.mod`) for the gin task.
211222

212223
`benchmarks/b7_task_correctness/.work` (B12's own convention) is **not** used
213224
here — see Isolation above; work copies land in `../calm-b7-work/` instead,
214225
which is not committed and safe to delete between runs.
215226

216227
## Next steps
217228

218-
1. **gin/Go** once a Go toolchain is available (Phase 2 completion).
219-
2. **spring-petclinic/Java** (Phase 3 — Maven+JVM, expected to be the
229+
1. **spring-petclinic/Java** (Phase 3 — Maven+JVM, expected to be the
220230
heaviest/flakiest setup of the five, per the design spec).
221-
3. Investigate the express `setCharset` call-graph gap directly in
231+
2. Investigate the express `setCharset` call-graph gap directly in
222232
`parser.rs`'s JS/TS call-site extraction (property-access call through a
223233
required module's bare identifier vs. a destructured bare-name call to
224234
the same export) — a candidate root-cause worth its own session, not

benchmarks/b7_task_correctness/run_benchmark.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@
7070
# needed to worry about for its own read-only use case.
7171
WORK_ROOT = repo_root_from_here().parent / "calm-b7-work"
7272

73-
SRC_EXTS = {"rust": (".rs",), "python": (".py",), "javascript": (".js",), "typescript": (".ts",)}
73+
SRC_EXTS = {"rust": (".rs",), "python": (".py",), "javascript": (".js",), "typescript": (".ts",), "go": (".go",)}
7474

7575

7676
def fresh_clone(lang: str, arm: str) -> Path:

benchmarks/lib/refactor_tasks.yaml

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -64,12 +64,37 @@ tasks:
6464
# removed upstream).
6565
test_cmd: ["uv", "run", "pytest", "-q"]
6666

67-
# --- Phase 2 (2026-07-30): Go was skipped this round -- `go` is not
68-
# installed in this environment and passwordless sudo is unavailable to
69-
# install it (verified live: `apt-cache policy golang-go` shows a
70-
# candidate but `sudo -n true` fails) -- a real, honestly-reported
71-
# environment precondition gap, not a benchmark bug. gin/Go is the
72-
# documented next addition once a Go toolchain is available.
67+
# --- Phase 2 continued: Go toolchain installed later the same day (user
68+
# provided sudo access specifically to unblock this) -- gin/Go added below.
69+
70+
- id: rename_gin_clean_path
71+
lang: go
72+
corpus: go # key into b12_tier1_tier2_tool_correctness.corpora.get_corpus()
73+
symbol: cleanPath
74+
def_path: path.go
75+
def_line: 23
76+
new_name: normalizeUrlPath
77+
# verified live 2026-07-30 against gin @ 34dac209ffb6ef85cc78c5d217bbb7ad001d68fd.
78+
# Same package (no cross-package qualification needed -- Go doesn't have
79+
# JS's require()-property-access call shape, so this cleanly isolates
80+
# the rename question from the property-access-call finding seen in
81+
# express/setCharset). Real production usage (gin.go, 2 sites) + heavy
82+
# test coverage (path_test.go, 6 sites) + 1 doc-comment mention
83+
# (path.go:10, included deliberately -- a thorough rename should update
84+
# doc comments too, even though the build doesn't strictly require it).
85+
oracle_callsites:
86+
- [gin.go, 704]
87+
- [gin.go, 812]
88+
- [path.go, 10]
89+
- [path_test.go, 74]
90+
- [path_test.go, 75]
91+
- [path_test.go, 89]
92+
- [path_test.go, 99]
93+
- [path_test.go, 130]
94+
- [path_test.go, 131]
95+
- [path_test.go, 142]
96+
build_cmd: ["go", "build", "./..."]
97+
test_cmd: ["go", "test", "./..."]
7398

7499
- id: rename_express_set_charset
75100
lang: javascript

0 commit comments

Comments
 (0)