Skip to content

Commit dcef3d5

Browse files
author
Your Name
committed
fix(indexer): B7 Phase 3 -- close Java this.field.method() call-graph blind spot
last_ident_segment only split a call receiver's text on `->`/`::`, never on a plain `.`. Java's method_invocation has no `->`, so `this.types.findPetTypes()`'s object-field text "this.types" resolved to receiver "this" instead of "types" -- tier-2 then looked up a type for the fake pseudo-variable "this", found nothing, and silently dropped the call edge. This made edit_context/callers() blind to every this.field.method() call in Java, reproduced live via B7's new spring-petclinic task (4 of 5 real call sites missed, CALM's own scripted rename arm failed the corpus's build where the naive grep arm passed). Fixed by adding `.` as a third segment separator; added a regression test. Also completes B7 Phase 3 (all 6 Tier-0 languages now covered) and updates the benchmark docs/results accordingly.
1 parent 388f7fd commit dcef3d5

5 files changed

Lines changed: 183 additions & 11 deletions

File tree

benchmarks/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ thật; khi số đo ra ngoài kỳ vọng (vd B6 `find_callers` = 0%), báo cá
1313
| B4 | Token Efficiency | MCP tools vs `cat`/`grep` naive workflow | **Implemented**[`b4_token_efficiency/`](b4_token_efficiency/) |
1414
| B5 | Incremental Indexing Speed | Reindex chỉ file thay đổi | Planned |
1515
| B6 | Tool-Call Efficiency | Số round-trip naive vs 1 MCP call (ý tưởng từ CodeGraph) | **Implemented**[`b6_tool_call_efficiency/`](b6_tool_call_efficiency/) |
16-
| B7 | Task Correctness / Regression | Refactor thật (rename) trên corpus thật per-language (fd/Rust, flask/Python), oracle xác định (build/test pass + callsite recall độc lập, không LLM-judge), đếm callsite bị bỏ sót (ý tưởng từ Serena) | **Implemented (Phase 1)**[`b7_task_correctness/`](b7_task_correctness/) |
16+
| B7 | Task Correctness / Regression | Refactor thật (rename) trên corpus thật per-language (fd/Rust, flask/Python, express/JS, zod/TS, gin/Go, spring-petclinic/Java), oracle xác định (build/test pass + reference recall độc lập, không LLM-judge), đếm reference bị bỏ sót (ý tưởng từ Serena) | **Implemented (Phase 1-3, all 6 languages)**[`b7_task_correctness/`](b7_task_correctness/) |
1717
| B8 | Model-Tier Leveling | Model rẻ + calm tools vs model đắt không có tools, cùng task (ý tưởng từ GitNexus) | Planned |
1818
| B9 | Scaling Curve | Lợi thế `calm` co giãn theo quy mô repo (nhỏ → lớn) | Planned |
1919
| B10 | Real Competitor A/B | `calm` vs CodeGraph vs Semble — tool call thật trên cả 3 MCP server thật (không phải số tự báo cáo) | **Superseded by B11**[`b10_real_competitor_ab/`](b10_real_competitor_ab/) (giữ lại, xem B11 cho methodology đã fix) |

benchmarks/b7_task_correctness/README.md

Lines changed: 79 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS, gin/Go)
1+
# B7 — Task-Correctness benchmark (Phase 1: fd/Rust, flask/Python; Phase 2: express/JS, zod/TS, gin/Go; Phase 3: spring-petclinic/Java)
22

33
Measures whether the CALM-scripted refactor workflow (`edit_context`
44
rename at each real reference → `diff_impact`) completes a real rename task
@@ -28,6 +28,7 @@ directly:
2828
| javascript | express (expressjs/express) | `npm install` | `npm test` |
2929
| typescript | zod (colinhacks/zod) | `pnpm install` | `pnpm test` |
3030
| go | gin (gin-gonic/gin) | `go build ./...` | `go test ./...` |
31+
| java | spring-petclinic (spring-projects) | *(none — `mvn test` compiles+tests in one step)* | `./mvnw -q test -Dtest='!*IntegrationTests' -DfailIfNoTests=false` |
3132

3233
**Go/gin was initially skipped** in this session's first Phase-2 pass — `go`
3334
wasn't installed and passwordless `sudo` was unavailable (verified live:
@@ -53,6 +54,17 @@ go.mod and guessing:**
5354
`GOTOOLCHAIN=auto` (Go's default since 1.21) transparently downloaded
5455
1.25.0 on first `go build`, no manual intervention needed. Verified by
5556
actually running the build, not assumed from the version mismatch alone.
57+
- spring-petclinic's Maven Wrapper + `~/.m2` cache were already warm from an
58+
earlier session's language-support benchmarking, so no network/toolchain
59+
install was needed for Java. The task's `test_cmd` excludes the 4
60+
Testcontainers-backed `*IntegrationTests` classes (they spin up real Docker
61+
MySQL/Postgres containers): a timed run including them took 2m04s on a cold
62+
image cache alone, real risk against `oracle.py::run_cmd`'s fixed 300s
63+
timeout given B7 runs build/test up to 3× per task (baseline + naive + calm
64+
arms). Excluding them costs zero oracle coverage — verified none of the 8
65+
real reference sites fall in those 4 files. `-DfailIfNoTests=false` is
66+
needed because `-Dtest` exclusion-only patterns otherwise make surefire
67+
treat "0 explicitly-matched classes" as a failure.
5668

5769
## Isolation
5870

@@ -94,7 +106,7 @@ building this, in two separate iterations:
94106
by widening the pattern to match the bare identifier regardless of what
95107
follows it.
96108

97-
## Results (all 5 tasks, current methodology)
109+
## Results (all 6 tasks, current methodology)
98110

99111
| task | baseline | naive build_pass | naive recall | naive tool_calls | calm build_pass | calm recall | calm tool_calls |
100112
|---|---|---|---|---|---|---|---|
@@ -103,6 +115,11 @@ building this, in two separate iterations:
103115
| rename_gin_clean_path | green | True | 1.0 | 3 | True | 1.0 | 4 |
104116
| rename_express_set_charset | green | True | 1.0 | 4 | **False** | **0.667** | 4 |
105117
| rename_zod_prettify_error | green | True | 1.0 | 5 | **False** | **0.5** | 4 |
118+
| rename_petclinic_find_pet_types | green | True | 1.0 | 7 | True | 1.0 | 4 |
119+
120+
Java's row reflects the state **after** the parser fix below — see Phase 3
121+
for the real bug this run found and fixed live, before this number was
122+
reachable.
106123

107124
### Phase 1 + gin (fd, flask, gin): an honest tie
108125

@@ -168,6 +185,55 @@ in `parser.rs`'s JS/TS call-site extraction (property-access calls on a
168185
required module's bare identifier vs. a destructured bare-name call to the
169186
same export).
170187

188+
### Phase 3 (spring-petclinic/Java): a real call-graph bug found and fixed
189+
live — `this.field.method()` calls were completely invisible
190+
191+
`findPetTypes` (`PetTypeRepository` interface method) was picked for the same
192+
reasons as gin's `cleanPath`: a single, unambiguous definition, called from 2
193+
production files and 3 test files with no name collision (ruled out
194+
`Owner.getPet`, 3 overloads, and `VetRepository.findAll`, which collides with
195+
`JpaRepository`'s own inherited `findAll`).
196+
197+
**First run: naive=1.0 recall/green build, calm=0.333 recall/`False` build —
198+
CALM's own arm broke its own rename.** `edit_context("findPetTypes")` (raw
199+
JSON) reported only 2 caller edges, both in `PetTypeFormatterTests.java`,
200+
missing `PetController.java` and `PetTypeFormatter.java` (**production code**)
201+
and `PetControllerTests.java`/`ClinicServiceTests.java`. The naive build
202+
failure output pinpointed exactly what didn't get renamed:
203+
`cannot find symbol: method findPetTypes()`.
204+
205+
**Root cause, found by reading the actual call sites, not guessing:** every
206+
missed site calls `this.types.findPetTypes()` (`this.`-qualified field
207+
access); the one call CALM *did* find is the only site written as a bare
208+
`types.findPetTypes()` (no `this.`). That 100%-correlated split led straight
209+
to `crates/calm-core/src/indexer/parser.rs::last_ident_segment` — the
210+
function that extracts a call's receiver from Java's `method_invocation`
211+
"object" field text. It split on `->` (PHP) and `::` (Rust/PHP scope) but
212+
**never on a plain `.`** — so for the object text `"this.types"`,
213+
`leading_ident` walked from byte 0 and stopped at the first `.`, returning
214+
`"this"` instead of `"types"`. Tier-2 resolution then looked up a declared
215+
type for the fake pseudo-variable `"this"`, found nothing, and silently
216+
dropped the call edge — no fallback, no low-confidence edge, nothing. This
217+
makes CALM's call graph (and `edit_context`/`callers()` built on it) blind to
218+
**every** `this.field.method()` call in Java — one of the two idiomatic
219+
field-access styles, used specifically to disambiguate a field from a
220+
same-named constructor parameter (`this.types = types;`, exactly
221+
`PetTypeRepository`'s own consumers' pattern, and a very common Spring/
222+
enterprise-Java convention).
223+
224+
**Fixed** by adding `.` as a third segment separator (alongside `->`/`::`),
225+
taking whichever separator occurs last in the text — the same "rightmost
226+
identifier segment" contract the function already documented, just extended
227+
to the one separator style it was missing. Verified additive, not a behavior
228+
change: PHP's `$this->helper` (no `.` in that text) and every existing
229+
`this`-only receiver test (`this.logIt()`, no `.` after `this`) are
230+
unaffected — confirmed by the full workspace test suite staying green (868
231+
passed) plus a new regression test
232+
(`test_java_this_qualified_field_call_produces_receiver_not_this`). Re-running
233+
the benchmark after the fix (and a `cargo build --release -p calm-cli` to
234+
pick it up): calm build_pass=`True`, recall=1.0, matching naive — see the
235+
Results table above.
236+
171237
### A rejected candidate, kept for the audit trail
172238

173239
`slugify` (`packages/zod/src/v4/core/util.ts:347`) was tried first and
@@ -204,6 +270,12 @@ original read-only tool-correctness use in B12).
204270
`crates/calm-server/src/tools/guardrails.rs`). A naive `c.get("path")`
205271
would have silently returned `None` for every caller.
206272
6. **`slugify` name-collision task pick** — see "A rejected candidate" above.
273+
7. **`last_ident_segment` never split on a plain `.`** (Phase 3, spring-
274+
petclinic's `findPetTypes`) — a real, previously-undiscovered call-graph
275+
gap in `crates/calm-core/src/indexer/parser.rs`, not a benchmark-harness
276+
bug like 1/2 above. See "Phase 3" above for the full root-cause. Fixed in
277+
`last_ident_segment` + a new regression test
278+
(`test_java_this_qualified_field_call_produces_receiver_not_this`).
207279

208280
## Running it
209281

@@ -226,10 +298,11 @@ which is not committed and safe to delete between runs.
226298

227299
## Next steps
228300

229-
1. **spring-petclinic/Java** (Phase 3 — Maven+JVM, expected to be the
230-
heaviest/flakiest setup of the five, per the design spec).
231-
2. Investigate the express `setCharset` call-graph gap directly in
301+
1. Investigate the express `setCharset` call-graph gap directly in
232302
`parser.rs`'s JS/TS call-site extraction (property-access call through a
233303
required module's bare identifier vs. a destructured bare-name call to
234304
the same export) — a candidate root-cause worth its own session, not
235-
folded into this benchmark's scope.
305+
folded into this benchmark's scope. (Phase 3's Java finding was this same
306+
shape of bug — receiver misattribution in `parser.rs` — so it's worth
307+
checking whether the `.`-split fix incidentally helps here too before
308+
assuming a separate root cause; not verified either way yet.)

benchmarks/b7_task_correctness/run_benchmark.py

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
#!/usr/bin/env python3
2-
"""B7 -- Task-Correctness benchmark (Phase 1: fd [Rust], flask [Python]).
2+
"""B7 -- Task-Correctness benchmark (Phase 1: fd [Rust], flask [Python];
3+
Phase 2: express [JS], zod [TS], gin [Go]; Phase 3: spring-petclinic [Java]).
34
45
Measures whether the CALM-scripted refactor workflow (edit_context -> edit at
56
each real call site -> diff_impact) completes a real rename task more
@@ -70,7 +71,7 @@
7071
# needed to worry about for its own read-only use case.
7172
WORK_ROOT = repo_root_from_here().parent / "calm-b7-work"
7273

73-
SRC_EXTS = {"rust": (".rs",), "python": (".py",), "javascript": (".js",), "typescript": (".ts",), "go": (".go",)}
74+
SRC_EXTS = {"rust": (".rs",), "python": (".py",), "javascript": (".js",), "typescript": (".ts",), "go": (".go",), "java": (".java",)}
7475

7576

7677
def fresh_clone(lang: str, arm: str) -> Path:
@@ -230,7 +231,7 @@ def main() -> int:
230231
rows.append(row)
231232

232233
summary = {
233-
"phase": "B7 Phase 1 (fd/Rust, flask/Python)",
234+
"phase": "B7 Phase 1-3 (fd/Rust, flask/Python, express/JS, zod/TS, gin/Go, spring-petclinic/Java)",
234235
"methodology": "deterministic oracle only (build/test pass + independent "
235236
"callsite recall via B12's ground_truth, extension-filtered) "
236237
"-- no LLM judge, per design spec constraint",

benchmarks/lib/refactor_tasks.yaml

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,3 +153,52 @@ tasks:
153153
- [packages/zod/src/v4/classic/tests/error-utils.test.ts, 544]
154154
build_cmd: ["pnpm", "install"]
155155
test_cmd: ["pnpm", "test"]
156+
157+
# --- Phase 3: spring-petclinic/Java (Maven+JVM) -- deferred to last since
158+
# it's the heaviest/flakiest setup per the design spec (§5 sequencing
159+
# table): confirmed live this session, Maven Wrapper + ~/.m2 cache were
160+
# already warm from an earlier session's language-support benchmarking, so
161+
# no network/toolchain install was needed here (unlike Go, which needed a
162+
# real sudo-gated toolchain install in Phase 2).
163+
- id: rename_petclinic_find_pet_types
164+
lang: java
165+
corpus: java # key into b12_tier1_tier2_tool_correctness.corpora.get_corpus()
166+
symbol: findPetTypes
167+
def_path: src/main/java/org/springframework/samples/petclinic/owner/PetTypeRepository.java
168+
def_line: 37
169+
new_name: findAllPetTypes
170+
# verified live 2026-07-30 against spring-petclinic @ 51045d1648dad955df586150c1a1a6e22ef400c2.
171+
# Single interface method (no overload -- ruled out Owner.getPet, which
172+
# has 3 overloads, and VetRepository.findAll, which collides with
173+
# JpaRepository's own inherited findAll), called from 2 production files
174+
# + 3 test files -- same real-usage-plus-test-coverage shape as gin's
175+
# cleanPath. NOTE (documented, not a rejected-candidate case unlike
176+
# zod's slugify): PetTypeFormatter.java:53 also declares a LOCAL
177+
# VARIABLE literally named `findPetTypes` (`Collection<PetType>
178+
# findPetTypes = this.types.findPetTypes();`), so the word-bounded
179+
# substitution renames that local too. This is benign, not a collision:
180+
# Java resolves the method call via `.name()` call-shape independently
181+
# of the local var's plain identifier, so renaming both consistently
182+
# within the same method body changes no semantics and both arms still
183+
# compile -- confirmed by the calm arm's real build/test pass below.
184+
oracle_callsites:
185+
- [src/main/java/org/springframework/samples/petclinic/owner/PetController.java, 63]
186+
- [src/main/java/org/springframework/samples/petclinic/owner/PetTypeFormatter.java, 53]
187+
- [src/main/java/org/springframework/samples/petclinic/owner/PetTypeFormatter.java, 54]
188+
- [src/test/java/org/springframework/samples/petclinic/owner/PetControllerTests.java, 72]
189+
- [src/test/java/org/springframework/samples/petclinic/owner/PetTypeFormatterTests.java, 65]
190+
- [src/test/java/org/springframework/samples/petclinic/owner/PetTypeFormatterTests.java, 72]
191+
- [src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java, 147]
192+
- [src/test/java/org/springframework/samples/petclinic/service/ClinicServiceTests.java, 166]
193+
build_cmd: null # `mvn test` compiles+tests in one step, same shape as Rust's `cargo test`
194+
# Excludes the 4 Testcontainers-backed *IntegrationTests classes (Docker
195+
# MySQL/Postgres containers) -- verified live: including them pulls a
196+
# ~260MB mysql image on a cold cache and pushes a single run past 2
197+
# minutes, unnecessary risk against oracle.py's fixed 300s run_cmd
198+
# timeout given B7 runs build/test up to 3x per task (baseline + naive +
199+
# calm arms). The excluded classes never reference findPetTypes (verified
200+
# against the oracle_callsites above), so excluding them costs zero
201+
# oracle coverage. `-DfailIfNoTests=false` needed because -Dtest exclusion
202+
# patterns alone make surefire treat "0 classes newly excluded to explicit
203+
# positive match" as a failure otherwise.
204+
test_cmd: ["./mvnw", "-q", "test", "-Dtest=!*IntegrationTests", "-DfailIfNoTests=false"]

crates/calm-core/src/indexer/parser.rs

Lines changed: 50 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1802,11 +1802,36 @@ fn walk_calls(
18021802
/// nested property access used as another call's receiver), "Foo" from a
18031803
/// bare "Foo" scope. Strips a leading `$` (PHP variable sigil) since
18041804
/// tier-2 type_map lookup keys on the plain variable name either way.
1805+
///
1806+
/// Also splits on a plain `.` (not just `->`/`::`) — found live via a real
1807+
/// B7 benchmark run (spring-petclinic/Java, docs/superskills/specs/2026-07-
1808+
/// 30-calm-dfb-levers-design.md Phase 3): Java's `method_invocation` has no
1809+
/// `->` operator, so `this.types.findPetTypes()`'s "object" field text is
1810+
/// the plain-dotted `"this.types"` — before this fix, the missing `.` split
1811+
/// point meant `leading_ident` walked from byte 0 and stopped at the first
1812+
/// `.`, returning `"this"` instead of the real field name `"types"`. Tier-2
1813+
/// then looked up a type for the fake pseudo-variable `"this"`, found
1814+
/// nothing, and dropped the call edge entirely — silently blind to EVERY
1815+
/// `this.field.method()` call in Java, one of the two idiomatic field-access
1816+
/// styles (used specifically to disambiguate a field from a same-named
1817+
/// constructor parameter, e.g. `this.types = types;`) and reproduced on 4 of
1818+
/// 5 real call sites in the corpus's own production+test code. Confirmed
1819+
/// this is genuinely additive, not a behavior change for existing correct
1820+
/// cases: PHP's `$this->helper` already resolved correctly via the `->`
1821+
/// split point (unaffected, no `.` in that text), and Java/PHP/Ruby's
1822+
/// existing single-segment `this`-only tests (`this.logIt()`) have no `.`
1823+
/// after `this` either, so `dot_end` never fires for them.
18051824
fn last_ident_segment(raw: &str) -> Option<String> {
18061825
let raw = raw.trim();
18071826
let arrow_end = raw.rfind("->").map(|i| i + 2);
18081827
let colon_end = raw.rfind("::").map(|i| i + 2);
1809-
let start = arrow_end.into_iter().chain(colon_end).max().unwrap_or(0);
1828+
let dot_end = raw.rfind('.').map(|i| i + 1);
1829+
let start = arrow_end
1830+
.into_iter()
1831+
.chain(colon_end)
1832+
.chain(dot_end)
1833+
.max()
1834+
.unwrap_or(0);
18101835
leading_ident(raw[start..].trim_start_matches('$'))
18111836
}
18121837
/// Extract call sites from a source file, each attributed to its enclosing function.
@@ -3278,6 +3303,30 @@ interface Shape {}
32783303
assert_eq!(find(&symbols, "Shape").kind, SymbolKind::Interface);
32793304
}
32803305

3306+
#[test]
3307+
fn test_java_this_qualified_field_call_produces_receiver_not_this() {
3308+
// Regression for the real bug B7's Phase 3 benchmark run found live
3309+
// (spring-petclinic/Java, docs/superskills/specs/2026-07-30-calm-
3310+
// dfb-levers-design.md): `this.field.method()` is one of Java's two
3311+
// idiomatic field-access styles (used to disambiguate a field from a
3312+
// same-named constructor parameter, e.g. `this.types = types;`).
3313+
// Before the `last_ident_segment` fix, the "object" field text
3314+
// "this.types" had no `.` split point, so `leading_ident` returned
3315+
// "this" instead of "types" -- tier-2 then looked up a type for the
3316+
// fake pseudo-variable "this", found nothing, and the call edge was
3317+
// dropped entirely (reproduced on 4 of 5 real call sites in the
3318+
// corpus, CALM's own edit_context missing every production caller).
3319+
let code = "class Repo {\n java.util.List<String> findAll() { return null; }\n}\nclass Consumer {\n private final Repo repo;\n Consumer(Repo repo) { this.repo = repo; }\n void use() {\n this.repo.findAll();\n }\n}\n";
3320+
let calls = extract_calls(code, "java", "Test.java").unwrap();
3321+
assert!(
3322+
calls.iter().any(|c| c.enclosing_name == "use"
3323+
&& c.callee == "findAll"
3324+
&& c.receiver.as_deref() == Some("repo")),
3325+
"this.field.method() should attribute the field name as receiver, \
3326+
not the literal string \"this\": {calls:?}"
3327+
);
3328+
}
3329+
32813330
#[test]
32823331
fn test_java_enum_record_constructor_kinds() {
32833332
let code = r#"

0 commit comments

Comments
 (0)