|
5 | 5 | {"id":"DART-ZERO-CALL-EDGES","claim":"Dart call-graph extraction produces zero call edges -- documented as a deliberate scope cut (tree-sitter grammar has no node kind for call), not a bug","benchmark":"resolution","calm_commit":null,"competitor_versions":null,"corpus_commit":"dart-lang/args @ 7a2dfb5","harness_commit":null,"command":"see benchmarks/resolution/README.md Run section","raw_output_path":"benchmarks/resolution/README.md","canonical_result":"178 symbols indexed, 0 call edges","status":"superseded","supersedes":[],"superseded_by":"DART-C3-CALL-EDGE-EXTRACTION","last_reproduced":"2026-07-11","evidence_gap":"calm_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28) -- exactly the kind of gap this registry exists to stop happening going forward"} |
6 | 6 | {"id":"DART-C3-CALL-EDGE-EXTRACTION","claim":"Dart call-edge extraction (C3) recovers real edges via a dedicated dart_call_from_member_access parser branch; the original 'no node kind for call' assumption was wrong -- Dart's grammar shares member_access/selector/argument_part nodes between calls and plain field access, distinguished only by whether the trailing selector wraps an argument_part","benchmark":"resolution","calm_commit":null,"competitor_versions":null,"corpus_commit":"dart-lang/args @ 7a2dfb5 (same corpus commit as the superseded measurement)","harness_commit":null,"command":"see benchmarks/resolution/README.md Run section","raw_output_path":"benchmarks/resolution/README.md","canonical_result":"178 symbols, 2166 call edges: 7.8% resolved, 11.2% textual, 80.9% ambiguous, 0% formal/inferred (no SCIP provider for Dart)","status":"current","supersedes":["DART-ZERO-CALL-EDGES"],"superseded_by":null,"last_reproduced":"2026-07-28","evidence_gap":"calm_commit not recorded in the original write-up; predates this checkout's available git history (starts 2026-07-28)"} |
7 | 7 | {"id":"B15-CROSS-LANG-COMPETITOR-AB","claim":"calm vs CodeGraph vs Ctxo vs Context+, file-recall on callers, all 6 Tier-0 languages (python/rust/go/javascript/typescript/java)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"822e238efc54df32da36505cf25890c2302ee06f","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"flask 36e4a824, fd 41532d11, gin 34dac209, express a3714473, zod 912f0f51, spring-petclinic 51045d16 -- see README Version pins section for full pins","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 68/72 (94.4%), codegraph 69/72 (95.8%), ctxo 23/24 (96.0%, java-only), contextplus 72/72 (100.0%)","status":"superseded","supersedes":[],"superseded_by":"B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX","last_reproduced":"2026-08-18","evidence_gap":"Ctxo go/js/ts excluded from aggregate -- setup verified real but query results structurally empty despite 3 rounds of harness verification; root-caused to a real, unresolved Ctxo-side persistence issue (see README), not fixed further within this session's scope. Superseded because the calm numbers themselves were measuring 2 real, since-fixed bugs, not calm's steady-state capability -- see superseding entry."} |
8 | | -{"id":"B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX","claim":"Same B15 A/B, rerun after fixing 2 real bugs the first run's own investigation found: a benchmark oracle regex false-positive (Java control-flow lines wrongly excluded as call sites) and a real calm core-resolver bug (inherited-method calls through a statically-typed receiver silently dropped instead of falling back to ambiguous)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"see git log after 0d9aa3a for the pipeline.rs inheritance-fallback fix + regression test, not yet committed as of this entry's last_reproduced date","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"same 6 pinned corpora as B15 (see README Version pins) -- java resample differs because the fixed oracle changes which symbols pass sample_symbols' filters, not because of cherry-picking","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 72/73 (98.6%), codegraph 69/73 (94.5%), ctxo 23/25 (92.0%, java-only), contextplus 73/73 (100.0%) -- calm's sole remaining miss is the already-known JS new-expression gap, unrelated to the inheritance bug this run fixed","status":"current","supersedes":["B15-CROSS-LANG-COMPETITOR-AB"],"superseded_by":null,"last_reproduced":"2026-08-18","evidence_gap":"Same Ctxo go/js/ts exclusion as the superseded entry (unchanged, unrelated to this run's fixes). Core-resolver fix (pipeline.rs::resolve_sites_to_edges) verified via a new regression test plus full cargo test -p calm-core/calm-server (1234+395 tests, 0 regressions) and a live re-query against the real spring-petclinic corpus, but was NOT re-verified against a live re-run of B12/B13/B7's own Java rows, which share the same oracle bug and could show similar movement if rerun."} |
| 8 | +{"id":"B15R2-CROSS-LANG-COMPETITOR-AB-INHERITANCE-FIX","claim":"Same B15 A/B, rerun after fixing 2 real bugs the first run's own investigation found: a benchmark oracle regex false-positive (Java control-flow lines wrongly excluded as call sites) and a real calm core-resolver bug (inherited-method calls through a statically-typed receiver silently dropped instead of falling back to ambiguous)","benchmark":"b15_cross_lang_competitor_ab","calm_commit":"62b4df2 (pipeline.rs inheritance-fallback fix + regression test; the benchmark run itself was against the pre-commit working tree at the same content)","competitor_versions":"@colbymchenry/codegraph@1.5.0, @ctxo/cli@0.11.4, contextplus@1.0.8","corpus_commit":"same 6 pinned corpora as B15 (see README Version pins) -- java resample differs because the fixed oracle changes which symbols pass sample_symbols' filters, not because of cherry-picking","harness_commit":null,"command":"see benchmarks/b15_cross_lang_competitor_ab/README.md Run section","raw_output_path":"benchmarks/b15_cross_lang_competitor_ab/README.md","canonical_result":"aggregate file-recall (Ctxo counts java only, see README disclosure): calm 72/73 (98.6%), codegraph 69/73 (94.5%), ctxo 23/25 (92.0%, java-only), contextplus 73/73 (100.0%) -- calm's sole remaining miss is the already-known JS new-expression gap, unrelated to the inheritance bug this run fixed","status":"current","supersedes":["B15-CROSS-LANG-COMPETITOR-AB"],"superseded_by":null,"last_reproduced":"2026-08-18","evidence_gap":"Same Ctxo go/js/ts exclusion as the superseded entry (unchanged, unrelated to this run's fixes). Core-resolver fix (pipeline.rs::resolve_sites_to_edges) verified via a new regression test plus full cargo test -p calm-core/calm-server (1234+395 tests, 0 regressions) and a live re-query against the real spring-petclinic corpus, but was NOT re-verified against a live re-run of B12/B13/B7's own Java rows, which share the same oracle bug and could show similar movement if rerun."} |
0 commit comments