Skip to content

Commit 297af80

Browse files
Your Nameclaude
andcommitted
fix(bench): Java oracle miscounts package-private test methods as call sites
`ground_truth.py`'s Java "method" definition pattern required an explicit `public`/`private`/`protected` modifier -- so a package-private method (the standard JUnit 5 convention for test methods, e.g. `void testFoo() { ... }` with no modifier at all) was never recognized as a *definition*. A JUnit test named after the production method it exercises (e.g. `void initUpdateOwnerForm() throws Exception` testing `OwnerController. initUpdateOwnerForm()`) fell through and got miscounted as a real *call site* of the production method. Found while building B15 (cross-language competitor benchmark): on spring-petclinic, CALM, CodeGraph, and Ctxo all scored 0/1 "missing" a call to `initUpdateOwnerForm()` that was never real -- the oracle's sole "hit" was the test method's own declaration line. Verified live: with the modifier now optional (same as the pre-existing class/interface patterns in this same file already treat it), `git_grep_call_sites` correctly returns 0 sites for that symbol. Shared by B7/B12/B13/B15 (all import `ground_truth.py`), so this closes the same false-positive class for every consumer, not just B15. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 822e238 commit 297af80

1 file changed

Lines changed: 14 additions & 1 deletion

File tree

benchmarks/b12_tier1_tier2_tool_correctness/ground_truth.py

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,20 @@ class Definition:
6060
(r"^\s*(?:public|private|protected)?\s*(?:static\s+)?(?:final\s+)?(?:abstract\s+)?class\s+(\w+)", "class"),
6161
(r"^\s*(?:public|private|protected)?\s*(?:static\s+)?(?:final\s+)?interface\s+(\w+)", "interface"),
6262
(
63-
r"^\s*(?:public|private|protected)\s+(?:static\s+)?(?:final\s+)?"
63+
# 2026-08-18 fix: the visibility modifier used to be MANDATORY
64+
# (`(?:public|private|protected)\s+`), so a package-private
65+
# method -- the standard JUnit 5 convention for test methods,
66+
# e.g. `void initUpdateOwnerForm() throws Exception {` with no
67+
# modifier at all -- was never recognized as a definition by
68+
# `_looks_like_a_definition`, and fell through to be counted as
69+
# a real CALL SITE of any production method sharing its name.
70+
# Verified live on spring-petclinic: a fresh B15 run scored
71+
# CALM/CodeGraph/Ctxo 0/1 "missing" OwnerController's
72+
# `initUpdateOwnerForm()` -- the sole "oracle file" was that
73+
# test method's OWN declaration line, not a real call anywhere.
74+
# Now optional, same as the class/interface patterns above
75+
# already treat it.
76+
r"^\s*(?:(?:public|private|protected)\s+)?(?:static\s+)?(?:final\s+)?"
6477
r"[\w<>\[\],\s]+?\s+(\w+)\s*\([^;{]*\)\s*(?:throws\s+[\w,\s]+)?\{",
6578
"method",
6679
),

0 commit comments

Comments
 (0)