Skip to content

Commit 095dd59

Browse files
Your Nameclaude
andcommitted
fix(resolver): WS7 — SCIP overlay must not add a formal edge conflicting with confident static resolution (D8)
Kills the corpus's one remaining false_confidence data point (fixture I / D8): false_confidence_rate 0.25 -> 0.0, false_confident_site_rate 0.125 -> 0.0, call_recall 0.875 (unchanged), no other fixture's outcome changed. MECHANISM CORRECTION (root-caused live, not from the audit or the fixture's own oracle note): the wrong `external.py::name@formal` edge is INSERTED by the SCIP overlay (formal_source='scip', insert_missing_exact_edges), NOT the stack-graphs `formally_resolved` bare-name upgrade in extract_file_data that the plan/audit assumed. Verified against the live DB: the call site is already `resolved`, so extract_file_data's formal upgrade (gated on `!= Resolved`) never fires here. scip-python follows the `from external import name` binding and reports external.py::name, missing that the later same-scope `def name` shadows it — so the overlay added a top-tier edge to a target real Python semantics never calls. Fix: insert_missing_exact_edges now skips inserting a competing formal/scip edge when the same call site already carries a CONFIDENT STATIC edge (`resolved`, or non-scip `formal`) to a DIFFERENT target (has_conflicting_confident_static_edge). Deliberately narrow: `ambiguous`/`textual`/`inferred` edges stay overridable — that IS the overlay's job (scip disambiguating fan-out) — only a real language-rule resolution is protected. The existing "scip rules out other targets of an ambiguous call site" behavior is unchanged (24/24 ingest tests, 1241/1241 calm-core lib tests green). Regression test (scip-independent, constructed directly): scip_does_not_add_formal_edge_conflicting_with_confident_static_resolution. The separate stack-graphs bare-name upgrade (pipeline.rs) remains a real but currently-undemonstrated latent false-confidence risk — tracked as follow-up, not fixed speculatively without a failing fixture. Wave 0.3 of docs/plans/2026-08-19-evidence-architecture-execution-plan.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent c10f72d commit 095dd59

3 files changed

Lines changed: 167 additions & 3 deletions

File tree

benchmarks/resolution_precision/README.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,34 @@ Implements **WS0** of
55
Every WS2–WS6 change in that plan must attach a before/after run of this benchmark; a change
66
merges only if `call_recall` does not drop **and** `false_confidence_rate` does not rise.
77

8+
## WS7 status: SHIPPED (2026-08-19) — fixture I / D8 false-confidence eliminated, `false_confidence_rate` 0.25 → 0.0
9+
10+
Fixture I (`I_file_symbol_wins_over_import`, the D8 shadowing case) used to emit TWO edges from one
11+
call site: `caller.py::name@resolved` (correct — tier-1 file-symbol-over-import priority) AND
12+
`external.py::name@formal` (WRONG). **Root-caused live (2026-08-19), correcting this fixture's own
13+
oracle note:** the wrong edge is inserted by the *SCIP overlay* (`formal_source = 'scip'`,
14+
`crates/calm-core/src/scip/ingest.rs::insert_missing_exact_edges`), NOT the bundled stack-graphs
15+
`formally_resolved` bare-name upgrade the plan/audit assumed. The call site's persisted confidence
16+
is already `resolved`, so `extract_file_data`'s formal upgrade (gated on `!= Resolved`) never fires
17+
here; scip-python follows the `from external import name` binding and reports `external.py::name`,
18+
missing that the later same-scope `def name` shadows it.
19+
20+
Fix: `insert_missing_exact_edges` now skips inserting a competing `formal`/`scip` edge when the same
21+
call site already carries a CONFIDENT STATIC edge (`resolved`, or non-scip `formal`) to a DIFFERENT
22+
target (`has_conflicting_confident_static_edge`). Deliberately narrow — `ambiguous`/`textual`/
23+
`inferred` edges stay overridable (that is the overlay's job); only a real language-rule resolution
24+
is protected. Regression test (scip-independent, constructed directly):
25+
`crates/calm-core/src/scip/ingest.rs::scip_does_not_add_formal_edge_conflicting_with_confident_static_resolution`.
26+
27+
Full corpus before → after: `false_confidence_rate` **0.25 → 0.0**, `false_confident_site_rate`
28+
0.125 → 0.0, `call_recall` 0.875 (unchanged), no other fixture's outcome changed; fixture I went
29+
FALSE_CONFIDENCE → RECALL_LOWCONF_CORRECT.
30+
31+
The separate stack-graphs `formally_resolved` bare-name upgrade (`pipeline.rs` — upgrades a site to
32+
`formal` when stack-graphs proved *any* same-named reference resolves in the file) is a real but
33+
currently-undemonstrated latent false-confidence risk; tracked as a follow-up, not fixed
34+
speculatively without a failing fixture.
35+
836
## WS2 status: SHIPPED (2026-08-18) — verify via unit test, not this corpus
937

1038
`import_path` now threads end-to-end (`resolve_tier1``CallSiteData``call_sites` column →

crates/calm-core/src/scip/ingest.rs

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -419,6 +419,21 @@ fn insert_missing_exact_edges(
419419
let Some(to_symbol) = resolve_unique_symbol_at(conn, def_path, *def_line)? else {
420420
continue;
421421
};
422+
// WS7 (evidence reconciliation, fixture I / D8): the static
423+
// resolver may have already resolved this EXACT call site to a
424+
// DIFFERENT target at a confident tier via a real language rule --
425+
// e.g. Python's file-symbol-over-import priority, where a later
426+
// same-scope `def name` shadows `from x import name`, so `name()`
427+
// never calls the import. SCIP is a strong authority but not an
428+
// infallible one: some SCIP indexers follow the import binding and
429+
// miss that shadowing. It must NOT silently ADD a competing
430+
// `formal` edge contradicting a confident static resolution -- that
431+
// is a genuine, live false_confidence_rate data point (a top-tier
432+
// edge to a target the language never actually calls). Skip the
433+
// insert on conflict; the correct static edge stays.
434+
if has_conflicting_confident_static_edge(conn, call_site_id, &to_symbol)? {
435+
continue;
436+
}
422437
let added = insert.execute(rusqlite::params![
423438
enclosing_qn,
424439
to_symbol,
@@ -446,6 +461,35 @@ fn insert_missing_exact_edges(
446461
Ok(inserted)
447462
}
448463

464+
/// WS7 reconciliation guard for `insert_missing_exact_edges`: true when this
465+
/// call site already carries a CONFIDENT STATIC edge to a target OTHER than the
466+
/// one SCIP wants to add. "Confident static" = `resolved` (tier-1 language-rule
467+
/// resolution) or a non-SCIP `formal` (e.g. stack-graphs) edge -- deliberately
468+
/// NOT `ambiguous`/`textual`/`inferred`, which the overlay is SUPPOSED to
469+
/// override (that is the whole point of SCIP disambiguating fan-out). A SCIP
470+
/// proof contradicting a confident static resolution is a conflict, not a new
471+
/// target: adding it would manufacture a false-confidence edge (D8).
472+
fn has_conflicting_confident_static_edge(
473+
conn: &Connection,
474+
call_site_id: i64,
475+
scip_target: &str,
476+
) -> rusqlite::Result<bool> {
477+
conn.query_row(
478+
"SELECT EXISTS(
479+
SELECT 1 FROM call_edges
480+
WHERE call_site_id = ?1
481+
AND to_symbol != ?2
482+
AND ruled_out_by_scip = 0
483+
AND ( edge_confidence = 'resolved'
484+
OR (edge_confidence = 'formal'
485+
AND (formal_source IS NULL OR formal_source != 'scip')) )
486+
)",
487+
rusqlite::params![call_site_id, scip_target],
488+
|row| row.get::<_, i64>(0),
489+
)
490+
.map(|exists| exists != 0)
491+
}
492+
449493
/// Persist evidence only after the graph row itself was accepted. The SELECT
450494
/// rechecks the current CallSite/span/file snapshot, so a stale or deleted
451495
/// edge cannot manufacture a proof record by id alone.
@@ -1397,6 +1441,88 @@ mod tests {
13971441
assert_eq!(source, "scip");
13981442
}
13991443

1444+
#[test]
1445+
fn scip_does_not_add_formal_edge_conflicting_with_confident_static_resolution() {
1446+
// WS7 / fixture I (D8): `from external import name; def name(): ...;
1447+
// name()`. Python's file-symbol-over-import priority correctly resolves
1448+
// the call to the LOCAL `name` at `resolved`. scip-python follows the
1449+
// import binding and reports external.py::name -- but real Python
1450+
// semantics never call it (the later same-scope def shadows the
1451+
// import). The overlay must NOT insert a competing `formal` edge that
1452+
// contradicts the confident static resolution (verified live 2026-08-19:
1453+
// it used to, producing a top-tier false_confidence edge).
1454+
let conn = Connection::open_in_memory().unwrap();
1455+
crate::db::schema::init_db(&conn).unwrap();
1456+
conn.execute_batch(
1457+
"INSERT INTO symbols (qualified_name, name, kind, language, path, line_start, line_end)
1458+
VALUES
1459+
('caller.py::name', 'name', 'function', 'python', 'caller.py', 4, 5),
1460+
('external.py::name', 'name', 'function', 'python', 'external.py', 1, 2);
1461+
INSERT INTO file_index (path, hash, language, symbol_count, last_indexed)
1462+
VALUES ('caller.py', 'fresh-source', 'python', 2, 0);
1463+
INSERT INTO call_sites
1464+
(from_path, enclosing_qn, callee_name, call_line, callee_start_byte,
1465+
callee_end_byte, identity_version, edge_kind)
1466+
VALUES ('caller.py', 'caller.py::use', 'name', 9, 4, 9, 2, 'call');
1467+
INSERT INTO call_edges
1468+
(from_symbol, to_symbol, call_site_line, call_site_id, edge_confidence,
1469+
from_path, to_path, edge_kind)
1470+
VALUES ('caller.py::use', 'caller.py::name', 9, 1, 'resolved',
1471+
'caller.py', 'caller.py', 'call');",
1472+
)
1473+
.unwrap();
1474+
let occ = vec![
1475+
crate::scip::parse::ScipOccurrence {
1476+
file: "external.py".into(),
1477+
line: 1,
1478+
start_byte: None,
1479+
end_byte: None,
1480+
source_file_hash: None,
1481+
symbol: "N".into(),
1482+
is_def: true,
1483+
is_local: false,
1484+
encoding_provenance: crate::scip::parse::EncodingProvenance::Declared,
1485+
guessed_alt_byte_range: None,
1486+
},
1487+
crate::scip::parse::ScipOccurrence {
1488+
file: "caller.py".into(),
1489+
line: 9,
1490+
start_byte: Some(4),
1491+
end_byte: Some(9),
1492+
source_file_hash: Some("fresh-source".into()),
1493+
symbol: "N".into(),
1494+
is_def: false,
1495+
is_local: false,
1496+
encoding_provenance: crate::scip::parse::EncodingProvenance::Declared,
1497+
guessed_alt_byte_range: None,
1498+
},
1499+
];
1500+
1501+
let stats = super::ingest_occurrences(&conn, &occ, true).unwrap();
1502+
assert_eq!(
1503+
stats.inserted, 0,
1504+
"scip must not insert a competing formal edge conflicting with the \
1505+
confident static (resolved) resolution of the same call site"
1506+
);
1507+
let edges: Vec<(String, String, i64)> = {
1508+
let mut stmt = conn
1509+
.prepare(
1510+
"SELECT to_symbol, edge_confidence, ruled_out_by_scip \
1511+
FROM call_edges ORDER BY to_symbol",
1512+
)
1513+
.unwrap();
1514+
stmt.query_map([], |r| Ok((r.get(0)?, r.get(1)?, r.get(2)?)))
1515+
.unwrap()
1516+
.collect::<rusqlite::Result<_>>()
1517+
.unwrap()
1518+
};
1519+
assert_eq!(
1520+
edges,
1521+
vec![("caller.py::name".to_string(), "resolved".to_string(), 0)],
1522+
"only the correct local edge should survive, un-ruled-out: {edges:?}"
1523+
);
1524+
}
1525+
14001526
#[test]
14011527
fn exact_span_reference_rules_out_only_other_targets_of_the_same_call_site() {
14021528
let conn = Connection::open_in_memory().unwrap();

docs/plans/2026-08-19-evidence-architecture-execution-plan.md

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,9 +111,19 @@ immediately.
111111
- **DoD:** `check-b2-thresholds.sh` passes on a corpus whose oracle coverage is
112112
measured and documented; thresholds reflect the honest baseline, not aspirational floors.
113113

114-
**0.3 — WS7A: preserve provider target identity** *(the P0 core; ~M)*
115-
- Change `formally_resolved_names` (`pipeline.rs:400-407`) to stop collapsing to
116-
`HashSet<String>`. Return the **pairs** `(reference_symbol, definition_symbol)`.
114+
**0.3 — WS7A: reconcile provider proof against confident static resolution** *(the P0 core; ~M)***SHIPPED 2026-08-19**
115+
> **Mechanism correction (verified live, not from the audit/fixture comment):** the D8
116+
> false-confidence edge is inserted by the **SCIP overlay**
117+
> (`scip/ingest.rs::insert_missing_exact_edges`, `formal_source='scip'`), *not* the
118+
> stack-graphs `formally_resolved` bare-name upgrade the audit and fixture I's own
119+
> oracle note blamed — the call site is already `resolved`, so `extract_file_data`'s
120+
> upgrade (gated on `!= Resolved`) never fires. Fix landed there instead:
121+
> `insert_missing_exact_edges` skips a competing `formal` insert when the same call
122+
> site already has a confident static edge (`resolved`/non-scip `formal`) to a different
123+
> target (`has_conflicting_confident_static_edge`). The stack-graphs bare-name upgrade
124+
> is a *separate*, currently-undemonstrated latent risk — follow-up, not fixed here.
125+
- ~~Change `formally_resolved_names` to return `(reference_symbol, definition_symbol)` pairs~~
126+
(superseded — that path is not the D8 cause; see correction above).
117127
- At the upgrade site (`pipeline.rs:771-776`), map `definition_symbol` → the target
118128
`SymbolId` the static resolver *actually chose* for this call site, and compare:
119129
- **agree** → confirm `Formal` (today's happy path, now *justified*);

0 commit comments

Comments
 (0)