-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathclaims.registry.jsonl
More file actions
9 lines (9 loc) · 11.7 KB
/
Copy pathclaims.registry.jsonl
File metadata and controls
9 lines (9 loc) · 11.7 KB
1
2
3
4
5
6
7
8
9
{"id":"B10-EFFICIENCY-VS-CODEGRAPH-SEMBLE","claim":"calm shows a large token-cost advantage over CodeGraph v1.2.0 and Semble on 4 self-repo tasks","benchmark":"b10_real_competitor_ab","calm_commit":null,"competitor_versions":"CodeGraph v1.2.0, Semble (uvx-fetched, unpinned)","corpus_commit":"self-repo (CALM), N=1 run","harness_commit":null,"command":"benchmarks/.venv/bin/python benchmarks/b10_real_competitor_ab/run_benchmark.py","raw_output_path":"benchmarks/b10_real_competitor_ab/README.md","canonical_result":"read_one_function token ratio vs naive: calm 19.3x, CodeGraph 12.6x, Semble 86.6x","status":"superseded","supersedes":[],"superseded_by":"B11-EXTENDED-COMPETITOR-AB","last_reproduced":"2026-07-06","evidence_gap":"calm_commit/harness_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28); only find_callers had a correctness oracle, N=1 with no repeats -- see B11's own README for the full audit of methodology bugs this superseded"}
{"id":"B11-EXTENDED-COMPETITOR-AB","claim":"calm vs CodeGraph vs Semble vs grepai vs Serena, all 5 B10 methodology bugs fixed (per-task correctness oracle, N=5 repeats, disclosed tokenizer proxy, warm-up readiness gate, new risk_gate_refusal/memory_recall tasks)","benchmark":"b11_extended_competitor_ab","calm_commit":null,"competitor_versions":"CodeGraph, Semble, grepai (Ollama nomic-embed-text), Serena (rust-analyzer-backed)","corpus_commit":"self-repo (CALM), Rust, N=5 repeats per task per tool","harness_commit":null,"command":"see benchmarks/b11_extended_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b11_extended_competitor_ab/README.md","canonical_result":"find_callers file-recall vs grep oracle (2 files): calm 2/2, CodeGraph 1/2, Semble/grepai/Serena 2/2","status":"current","supersedes":["B10-EFFICIENCY-VS-CODEGRAPH-SEMBLE"],"superseded_by":null,"last_reproduced":"2026-07-06","evidence_gap":"calm_commit/harness_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28)"}
{"id":"B13-PHASE1-2-FILE-RECALL","claim":"calm vs CodeGraph file-recall, multi-repo (fd, flask, self-repo)","benchmark":"b13_codegraph_multirepo_ab","calm_commit":"aba60aa86b7215cdce755d12835083329f4c7172","competitor_versions":"@colbymchenry/codegraph@1.5.0","corpus_commit":"fd (sharkdp/fd), flask, self-repo -- see README for per-corpus pins","harness_commit":"aba60aa86b7215cdce755d12835083329f4c7172 (harness + calm from the same session, not a separate commit at benchmark time)","command":"see benchmarks/b13_codegraph_multirepo_ab/README.md Methodology section","raw_output_path":"benchmarks/b13_codegraph_multirepo_ab/README.md","canonical_result":"combined file-recall: calm 80.6%, CodeGraph 72.2%","status":"superseded","supersedes":[],"superseded_by":"B13-CORRECTED-FILE-RECALL","last_reproduced":"2026-08-02","evidence_gap":null}
{"id":"B13-CORRECTED-FILE-RECALL","claim":"calm vs CodeGraph file-recall, multi-repo (fd, flask, self-repo); fixes a ground-truth oracle bug (doc/comment lines miscounted as call sites) and a SCIP-overlay race (recall measured before the async formal-confidence upgrade finished), and along the way finds + fixes a real CALM parser bug (Self::method() calls resolved to zero edges)","benchmark":"b13_codegraph_multirepo_ab","calm_commit":"52d1abe682069e018d00d09f2d9ab07b820a557c plus the oracle/scip_refresh-race fixes, uncommitted at benchmark time and later committed as 9c0b0fe; the Self:: parser fix landed separately as c5823a8","competitor_versions":"@colbymchenry/codegraph@1.5.0","corpus_commit":"fd (sharkdp/fd) @ 41532d1, flask, self-repo -- see README for full pins","harness_commit":"9c0b0fe","command":"see benchmarks/b13_codegraph_multirepo_ab/README.md Methodology section","raw_output_path":"benchmarks/b13_codegraph_multirepo_ab/README.md","canonical_result":"combined file-recall, N=24 symbols / 31 oracle files: calm 31/31 (100%), CodeGraph 27/31 (87.1%)","status":"current","supersedes":["B13-PHASE1-2-FILE-RECALL"],"superseded_by":null,"last_reproduced":"2026-08-02","evidence_gap":null}
{"id":"DART-ZERO-CALL-EDGES","claim":"Dart call-graph extraction produces zero call edges -- documented as a deliberate scope cut (tree-sitter grammar has no node kind for call), not a bug","benchmark":"resolution","calm_commit":null,"competitor_versions":null,"corpus_commit":"dart-lang/args @ 7a2dfb5","harness_commit":null,"command":"see benchmarks/resolution/README.md Run section","raw_output_path":"benchmarks/resolution/README.md","canonical_result":"178 symbols indexed, 0 call edges","status":"superseded","supersedes":[],"superseded_by":"DART-C3-CALL-EDGE-EXTRACTION","last_reproduced":"2026-07-11","evidence_gap":"calm_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28) -- exactly the kind of gap this registry exists to stop happening going forward"}
{"id":"DART-C3-CALL-EDGE-EXTRACTION","claim":"Dart call-edge extraction (C3) recovers real edges via a dedicated dart_call_from_member_access parser branch; the original 'no node kind for call' assumption was wrong -- Dart's grammar shares member_access/selector/argument_part nodes between calls and plain field access, distinguished only by whether the trailing selector wraps an argument_part","benchmark":"resolution","calm_commit":null,"competitor_versions":null,"corpus_commit":"dart-lang/args @ 7a2dfb5 (same corpus commit as the superseded measurement)","harness_commit":null,"command":"see benchmarks/resolution/README.md Run section","raw_output_path":"benchmarks/resolution/README.md","canonical_result":"178 symbols, 2166 call edges: 7.8% resolved, 11.2% textual, 80.9% ambiguous, 0% formal/inferred (no SCIP provider for Dart)","status":"current","supersedes":["DART-ZERO-CALL-EDGES"],"superseded_by":null,"last_reproduced":"2026-07-28","evidence_gap":"calm_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28)"}
{"id":"B15-CROSS-LANG-COMPETITOR-AB","claim":"calm vs CodeGraph vs Ctxo vs Context+, file-recall on callers, all 6 Tier-0 languages (python/rust/go/javascript/typescript/java)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"822e238efc54df32da36505cf25890c2302ee06f","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"flask 36e4a824, fd 41532d11, gin 34dac209, express a3714473, zod 912f0f51, spring-petclinic 51045d16 -- see README Version pins section for full pins","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 68/72 (94.4%), codegraph 69/72 (95.8%), ctxo 23/24 (96.0%, java-only), contextplus 72/72 (100.0%)","status":"superseded","supersedes":[],"superseded_by":"B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX","last_reproduced":"2026-08-18","evidence_gap":"Ctxo go/js/ts excluded from aggregate -- setup verified real but query results structurally empty despite 3 rounds of harness verification; root-caused to a real, unresolved Ctxo-side persistence issue (see README), not fixed further within this session's scope. Superseded because the calm numbers themselves were measuring 2 real, since-fixed bugs, not calm's steady-state capability -- see superseding entry."}
{"id":"B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX","claim":"Same B15 A/B, rerun after fixing 2 real bugs the first run's own investigation found: a benchmark oracle regex false-positive (Java control-flow lines wrongly excluded as call sites) and a real calm core-resolver bug (inherited-method calls through a statically-typed receiver silently dropped instead of falling back to ambiguous)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"62b4df2 (pipeline.rs inheritance-fallback fix + regression test; the benchmark run itself was against the pre-commit working tree at the same content)","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"same 6 pinned corpora as B15 (see README Version pins) -- java resample differs because the fixed oracle changes which symbols pass sample_symbols' filters, not because of cherry-picking","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 72/73 (98.6%), codegraph 69/73 (94.5%), ctxo 23/25 (92.0%, java-only), contextplus 73/73 (100.0%) -- calm's sole remaining miss is the already-known JS new-expression gap, unrelated to the inheritance bug this run fixed","status":"superseded","supersedes":["B15-CROSS-LANG-COMPETITOR-AB"],"superseded_by":"B15R3-CROSS-LANG-COMPETITOR-AB-NEW-EXPRESSION-FIX","last_reproduced":"2026-08-18","evidence_gap":"Same Ctxo go/js/ts exclusion as the superseded entry (unchanged, unrelated to this run's fixes). Core-resolver fix (pipeline.rs::resolve_sites_to_edges) verified via a new regression test plus full cargo test -p calm-core/calm-server (1234+395 tests, 0 regressions) and a live re-query against the real spring-petclinic corpus, but was NOT re-verified against a live re-run of B12/B13/B7's own Java rows, which share the same oracle bug and could show similar movement if rerun. Superseded because calm's one remaining miss (javascript new-expression) was fixed the same day -- see superseding entry."}
{"id":"B15R3-CROSS-LANG-COMPETITOR-AB-NEW-EXPRESSION-FIX","claim":"Same B15 A/B, rerun after fixing calm's last remaining miss: new_expression/object_creation_expression (new Foo(...)) wasn't treated as a call site in JS/TS/Java, verified via the real vendored tree-sitter grammars' node-types.json, fixed via the existing call_function_field_by_kind extension point (data-only change, same pattern already used for PHP/Ruby/Kotlin/Swift/Dart)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"0f1dd1c (new_expression/object_creation_expression call_node_types fix + 3 regression tests; the benchmark run itself was against the pre-commit working tree at the same content)","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"same 6 pinned corpora as B15 (see README Version pins)","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 73/73 (100.0%), codegraph 69/73 (94.5%), ctxo 23/25 (92.0%, java-only), contextplus 73/73 (100.0%) -- calm reaches zero misses across all 48 sampled symbols in 6 languages, tied with Context+'s recall-maximizing (substring-matching) ceiling but via a real resolver, not text search","status":"current","supersedes":["B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX"],"superseded_by":null,"last_reproduced":"2026-08-18","evidence_gap":"Same Ctxo go/js/ts exclusion as prior entries (unchanged, unrelated). The equivalent new-expression gap for C++/C#/Kotlin/Swift was NOT fixed or even verified this session -- their tree-sitter grammar crates aren't fetched under this repo's default build features, so their node-types.json wasn't inspected; flagged as a likely-present, unconfirmed gap for a future pass. Java's fix also surfaced (not fixed) a pre-existing ambiguity in Java's own symbol model: a class and its own constructor are indexed as two separate same-named symbols, so a bare-name constructor-call resolution returns both as ambiguous candidates rather than one -- correct behavior given no narrowing signal exists, but worth knowing about."}