fix(consistency): honest caveat for ambiguous symbol misses, no verdict change

ADR-0018 flagged backtick-quoted local variable names (with_tax,
days_overdue) as a nuisance false-positive source: real names, just not
graph-indexed, but flagged CONTRADICTED same as a hallucinated function
name. Investigating a fix surfaced a harder fact: a real local variable
name and a real hallucinated function name are lexically identical --
plain snake_case, no dot, no parens, no capital letter. All 14 invented
names the pilot caught had exactly that shape. Any heuristic that softened
the verdict or confidence for that shape would have silently cost the
14/14 recall already measured and published.

Fix: check_symbol_exists now adds an honest caveat to the explanation text
when a missed symbol has no marker distinguishing it from an ordinary
variable name (new looks_like_bare_name() in claims.py: no dot, no call
parens, no capital) -- status and confidence stay exactly as before,
CONTRADICTED at 1.0 either way. The caveat necessarily appears on both
with_tax and get_tax_rate, since nothing in the text tells them apart --
stated plainly in the explanation and in ADR-0018, not hidden. A clearly
code-shaped miss (TotallyMadeUpClass, CamelCase) gets no caveat, since
there the evidence really is as strong as the confidence claims.

6 new regression tests (test_claims.py's TestLooksLikeBareName,
test_consistency_check.py's two new TestSymbolExistence cases). ADR-0018
updated to reflect the fix and the lexical-ambiguity finding behind it.

518 tests passing (512 + 6 new).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: Juanpacol

docs/adr/0018-consistency-engine-first-measurement.md

@@ -112,13 +112,23 @@ synthetic-fixture trap's blind spots.
   here. The extractor's other named gap (tolerating explanatory text
   between the relation verb and its arguments, e.g. "likely calls a helper
   in") remains open, deliberately out of scope for this pass.
-- **Backtick-quoted local variable names are a genuine nuisance-false-
-  positive source** in real usage — anyone writing a normal technical
-  summary with `` `some_var` `` for readability will trigger a false
-  contradiction today. Worth a narrower fix (distinguishing "this claims a
-  codebase symbol" from "this is emphasis") in a future pass; not attempted
-  here to keep this pilot's scope to measurement plus the one clear,
-  high-confidence bug it found.
+- **Backtick-quoted local variable names — addressed, deliberately without
+  changing any verdict** (same-day follow-up). Investigating a fix
+  surfaced a harder fact: a real local variable name (`with_tax`) and a
+  real hallucinated function name (`get_tax_rate`) are lexically
+  identical — plain snake_case, no dot, no parens, no capital letter. All
+  14 invented names this pilot caught had exactly that shape. Any
+  heuristic that softened the verdict or confidence for that shape would
+  have silently cost the 14/14 recall this ADR reports. Instead,
+  `check_symbol_exists` now adds an honest caveat to the *explanation*
+  when a missed symbol has no marker distinguishing it from a plain
+  variable name (`looks_like_bare_name` in `claims.py`) — status and
+  confidence are unchanged, `CONTRADICTED` at 1.0 either way. The caveat
+  necessarily appears on both `with_tax` and `get_tax_rate`, since nothing
+  in the text tells them apart; this is stated plainly rather than
+  pretended away. Regression tests confirm a clearly code-shaped miss
+  (`TotallyMadeUpClass`) gets no caveat, since there the evidence really is
+  as strong as the confidence claims.
 - **The resurfacing fix is a real improvement, not a complete solution.**
   BM25's IDF is inherently unstable with 1-2 documents; a small residual
   false-positive risk remains for very small decision corpora. Stated

src/verityai/consistency/check.py

@@ -24,7 +24,7 @@
 
 from pathlib import Path
 
-from verityai.consistency.claims import extract_claims, looks_like_path
+from verityai.consistency.claims import extract_claims, looks_like_bare_name, looks_like_path
 from verityai.context.rank import bm25_rank
 from verityai.core.models import (
     CheckStatus,
@@ -67,11 +67,21 @@ def check_symbol_exists(claim: Claim, query: GraphQuery) -> ClaimCheck:
                 for m in matches[:5]
             ],
         )
+    explanation = f"no definition of {claim.subject!r} found anywhere in the graph"
+    if looks_like_bare_name(claim.subject, claim.raw_text):
+        # Same status and confidence as any other miss -- the verdict does
+        # not change. Only the explanation gets more honest: this token's
+        # shape cannot be distinguished from an ordinary local variable
+        # name backtick-quoted for emphasis (ADR-0018).
+        explanation += (
+            " (note: this could also be a local variable name, which the graph "
+            "does not track, rather than a hallucinated symbol)"
+        )
     return ClaimCheck(
         claim=claim,
         status=CheckStatus.CONTRADICTED,
         confidence=1.0,
-        explanation=f"no definition of {claim.subject!r} found anywhere in the graph",
+        explanation=explanation,
     )
 
 

src/verityai/consistency/claims.py

@@ -90,6 +90,23 @@ def looks_like_path(token: str) -> bool:
 _looks_like_path = looks_like_path
 
 
+def looks_like_bare_name(subject: str, raw_text: str) -> bool:
+    """True if this token carries no marker distinguishing it from an
+    ordinary local variable name -- no call parens, no dotted/qualified
+    path, no CamelCase.
+
+    This is deliberately the *same* shape a hallucinated bare function name
+    has (ADR-0018's pilot caught 14/14 invented names, all in exactly this
+    shape: plain snake_case, no punctuation, no parens). There is no
+    lexical signal that separates "someone backtick-quoted a local variable
+    for emphasis" from "someone invented a function name" -- so this is a
+    caveat signal for the checker's explanation text, never a reason to
+    change a verdict or confidence. Doing that would have silently traded
+    away the very recall ADR-0018 measured.
+    """
+    return "." not in subject and "()" not in raw_text and not any(c.isupper() for c in subject)
+
+
 def _looks_like_symbol(token: str) -> bool:
     return bool(_SYMBOL.match(token)) and (
         "." in token or "_" in token or token.endswith("()") or any(c.isupper() for c in token[1:])

tests/unit/test_claims.py

@@ -8,7 +8,7 @@
 as the subject instead of the actual symbol.
 """
 
-from verityai.consistency.claims import extract_claims
+from verityai.consistency.claims import extract_claims, looks_like_bare_name
 from verityai.core.models import ClaimKind
 
 
@@ -133,6 +133,24 @@ def test_a_relation_match_is_not_also_extracted_as_two_bare_symbols(self):
         assert len(claims) == 1
 
 
+class TestLooksLikeBareName:
+    """A caveat signal only -- see check_symbol_exists. Never used to change
+    a verdict, since the same shape (plain snake_case) is exactly what
+    ADR-0018's real hallucinated function names looked like too."""
+
+    def test_plain_snake_case_is_a_bare_name(self):
+        assert looks_like_bare_name("with_tax", "`with_tax`") is True
+
+    def test_a_dotted_qualname_is_not_bare(self):
+        assert looks_like_bare_name("Service.run", "`Service.run`") is False
+
+    def test_a_call_marker_means_not_bare(self):
+        assert looks_like_bare_name("get_tax_rate", "`get_tax_rate()`") is False
+
+    def test_camel_case_is_not_bare(self):
+        assert looks_like_bare_name("SomeClass", "`SomeClass`") is False
+
+
 class TestNoExtraction:
     def test_empty_text_extracts_nothing(self):
         assert extract_claims("") == []

tests/unit/test_consistency_check.py

@@ -91,6 +91,28 @@ def test_a_qualified_method_name_resolves(self, query):
 
         assert result.status is CheckStatus.SUPPORTED
 
+    def test_a_bare_snake_case_miss_gets_a_caveat_not_a_softer_verdict(self, query):
+        """Regression (ADR-0018): `with_tax` backtick-quoted for emphasis in
+        ordinary prose is lexically identical to an invented function name.
+        The verdict must stay CONTRADICTED at full confidence -- softening
+        it would have silently cost the 14/14 recall the real pilot
+        measured -- but the explanation should say the evidence is weaker
+        here than for a clearly code-shaped miss."""
+        result = check_symbol_exists(symbol_claim("with_tax"), query)
+
+        assert result.status is CheckStatus.CONTRADICTED
+        assert result.confidence == 1.0
+        assert "local variable" in result.explanation
+
+    def test_a_clearly_code_shaped_miss_gets_no_caveat(self, query):
+        """`TotallyMadeUpClass` (CamelCase) isn't the shape a bare local
+        variable name would take, so the caveat would be misleading noise
+        here -- the evidence really is as strong as the confidence claims."""
+        result = check_symbol_exists(symbol_claim("TotallyMadeUpClass"), query)
+
+        assert result.status is CheckStatus.CONTRADICTED
+        assert "local variable" not in result.explanation
+
 
 class TestSymbolRelation:
     def test_a_real_relation_is_supported(self, query):