fix(consistency): decision resurfacing false-positives on small corpora; first real measurement

docs/BENCHMARK_PROTOCOL.md claimed hallucination/contradiction detection
was blocked on the Consistency Engine existing at all -- stale. The engine
(consistency/, ADR-0007) already existed and was already tested. The real
gap was the same one Context Engine had before ADR-0009: no measurement
against real, not hand-authored, input.

Part A: five real agent trials, each shown only billing/invoice.py (no
other file of a small billing package), asked to describe the full call
chain anyway -- reliably produces a genuine mix of true claims and
hallucinated ones about functions the agent can't see. Ground truth was
written from the real source before running any checker output, to avoid
biasing labels. Result: 100% recall (14/14) on invented function names,
but a confirmed, structural blind spot: a claim that a function "calls
into" a file (rather than another function) never parses as a relation
claim at all -- it silently decomposes into two independent, both-true
existence checks. Verified directly with isolated probes, not just
inferred from trial transcripts. Also found: backtick-quoted local
variable names (with_tax, days_overdue) get flagged as "no definition
found", since the graph indexes functions/classes/files, not local
variables -- a real nuisance-false-positive source in normal technical
writing.

Part B: a real, fixed bug. check_decision_resurfacing normalized each
candidate decision's BM25 score against the maximum score *among the
stored decisions*, not an absolute scale. With one or two decisions on
record -- the common case for an early project, not an edge case --
whichever decision is relatively closest to the checked text always
normalized to a perfect 1.0, regardless of real relevance. Confirmed by
checking a proposal about caching tax rates against a fabricated rejected
decision about late-fee policy: completely unrelated, still flagged at 85%
confidence; adding a second, obviously unrelated rejected decision
("plaintext passwords") got both flagged.

Fix: normalize against the checked text's own best-possible score (BM25
matched against itself) instead of the in-corpus max. Confidence dropped
from an always-maximal 85% to 16% for the unrelated proposal and to a more
proportionate 43% for a genuine paraphrase of the rejected decision -- both
directionally correct. A residual small-corpus false-positive risk remains
(BM25's IDF is inherently unstable with 1-2 documents), stated honestly in
the ADR rather than oversold. Regression test added -- the pre-existing
"unrelated text" test never caught this because its fixture had zero token
overlap with the stored decision, missing the near-zero-but-nonzero score
regime where the bug lived.

507 tests passing (506 + 1 new regression test). See
docs/adr/0018-consistency-engine-first-measurement.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Author: Juanpacol

README.md

@@ -260,6 +260,31 @@ larger trial budget) to move — while the cost effect is now this project's
 most consistently reproduced Family B result, across three separate
 pilots and bug designs.
 
+**The Consistency Engine got its own first real measurement**
+(`experiments/consistency_pilot_1_hallucination_detection/`, [ADR-0018](docs/adr/0018-consistency-engine-first-measurement.md)),
+closing a stale claim in `docs/BENCHMARK_PROTOCOL.md` that this engine was
+still "blocked on existing at all" — it already existed (ADR-0007) and just
+lacked a measurement against real, not hand-authored, claims. Real agents,
+shown only one file of a small codebase, were asked to describe it anyway,
+producing a genuine mix of true and hallucinated claims:
+
+| Claim class | Result |
+|---|---|
+| Invented function names (14 across 5 trials) | **100% caught** |
+| Function-to-file relation hallucinations (~6) | **0% caught** — structurally invisible to the relation extractor |
+| Backtick-quoted local variable names (8) | False positives — real names, just not graph-indexed |
+
+A real bug was also found and fixed in decision resurfacing: with only one
+or two rejected decisions on record, the closest-of-the-available matches
+always normalized to 100% confidence regardless of actual relevance — a
+genuinely unrelated proposal "resembled" an unrelated rejected decision just
+as strongly as an actual paraphrase of it. Confirmed with an isolated probe,
+fixed by normalizing against the checked text's own best-possible score
+instead of the in-corpus max, and locked in with a regression test. See
+ADR-0018 for the full write-up, including what's still an open, unfixed
+blind spot (the relation-extraction gap) versus what got fixed this pass
+(the resurfacing bug).
+
 ---
 
 ## Why this project changed shape

docs/BENCHMARK_PROTOCOL.md

@@ -50,6 +50,18 @@ series, 5/5 both conditions — but the cost effect reproduced a third time:
 limits (small N, narrow task design, single fixture) still apply — see each
 pilot's own README before generalizing any result.
 
+Separately, the Consistency Engine (`consistency/`, ADR-0007) got its own
+first real measurement (`experiments/consistency_pilot_1_hallucination_
+detection/`, ADR-0018) — not a Family A/B comparison, but the same
+"real data before publishing" discipline applied to a different engine.
+Real agents, shown only one file of a small codebase, produced a genuine
+mix of true and hallucinated claims about it. Result: 100% recall on
+invented symbol names (14/14 caught), a confirmed blind spot on
+function-to-file relation claims (0% caught, structurally invisible to the
+extractor), and a real bug in decision resurfacing — a corpus of one or two
+decisions always normalized its closest match to 100% confidence regardless
+of actual relevance — found, fixed, and regression-tested.
+
 ## The two families
 
 Metrics are split by whether an LLM is in the loop, and the two families are
@@ -126,8 +138,13 @@ rows are Family B, and the honest position today is:
 - **Tokens and cost per task**: Family A for the pipeline itself. Becomes
   Family B the moment task success enters the denominator, because a cheaper
   run that fails is not cheaper.
-- **Hallucinations, contradictions, regressions**: Family B, and additionally
-  blocked on the Consistency Engine existing at all (Phase 3).
+- **Hallucinations, contradictions, regressions**: Family B. The
+  Consistency Engine (`consistency/`, ADR-0007) exists and has a first real
+  measurement (ADR-0018): 100% recall on invented symbol names in real
+  agent-generated text, a confirmed blind spot on function-to-file relation
+  claims, and a real decision-resurfacing bug found and fixed. Not yet
+  measured: recall/precision at scale, or on hallucinations about relations
+  between two real symbols phrased loosely.
 - **Technical debt**: not yet operationally defined. No metric until it is.
 - **Recovery after reset**: Family B, and the most valuable row in the table.
   It is the one thing the harness does that an agent cannot do for itself.

docs/adr/0018-consistency-engine-first-measurement.md

@@ -0,0 +1,110 @@
+# ADR-0018: First real measurement of the Consistency Engine -- and a real bug it found
+
+- **Status**: Accepted
+- **Date**: 2026-08-09
+- **Context**: `docs/BENCHMARK_PROTOCOL.md` describes hallucination and
+  contradiction detection as blocked on "the Consistency Engine existing at
+  all (Phase 3)." Before planning that as new work, the actual state of
+  `src/verityai/consistency/` was checked: the engine already exists,
+  already has tests, and was accepted in ADR-0007. The real gap was a
+  measurement against genuine (not hand-authored) input — the same gap
+  Context Engine had before ADR-0009.
+
+## Part A: inducing real hallucinations
+
+Writing false claims by hand to test the checker would measure the
+fixture, not the checker — the synthetic-fixture trap this project has
+avoided since ADR-0009. Instead, five independent live-agent trials were
+shown only `billing/invoice.py` (from the pilot 5 fixture, with a real
+code graph built via `verity graph build`) and asked to describe the full
+call chain anyway, including functions they could not see. Ground truth was
+written from the real source files before running any checker output.
+
+### Result
+
+98 claims checked, 23 flagged:
+
+| Class | Count | Outcome |
+|---|---|---|
+| Invented helper-function names | 14 | 100% caught |
+| Backtick-quoted local variable names (`with_tax`, `days_overdue`) | 8 | False positives — real names, just not graph-indexed |
+| A genuine path inaccuracy | 1 | Correctly caught |
+| Fabricated function-to-file relation claims | ~6 | **0% caught — structurally invisible**, not even counted among the 98 |
+
+Confirmed with isolated probes, not just inferred: `` `apply_tax` calls
+`billing/tax_rates.py` `` produces zero contradictions (decomposes into two
+independent, both-true existence checks), while `` `apply_tax` calls
+`get_tax_rate` `` — an equally fabricated relation but with a function-shaped
+target — is correctly caught as a relation-level failure. The relation
+extractor (`consistency/claims.py`) only recognizes symbol-to-symbol
+`calls`/`inherits from`/`extends` phrases; a claim of a function calling
+into a *file* — a very natural way to describe a module dependency — never
+parses as a relation claim at all.
+
+## Part B: a real bug in decision resurfacing
+
+A `REJECTED` decision was fabricated, then two proposals were checked: a
+genuine paraphrase of it (should trigger resurfacing) and a completely
+unrelated one (should not). **Both triggered resurfacing.**
+
+Root cause, in `check_decision_resurfacing`
+(`src/verityai/consistency/check.py`): each candidate decision's BM25 score
+was normalized against `max(scores.values())` — the maximum *among the
+stored decisions*, not an absolute scale. With one or two decisions on
+record (not an edge case — the common case for an early or solo project),
+whichever decision is relatively closest to the checked text always
+normalizes to a perfect 1.0, regardless of whether it shares any real
+content with it. Confirmed directly: adding a second, obviously unrelated
+rejected decision ("store passwords in plaintext") caused *both* to be
+flagged against a caching proposal that resembled neither.
+
+### Fix
+
+Normalize against the checked text's own best-possible score (BM25-matched
+against itself) instead of the in-corpus max:
+
+```python
+_, self_scores = bm25_rank(text, [text])
+self_score = self_scores.get(0, 0.0)
+if self_score <= 0:
+    return []
+...
+normalized = score / self_score
+```
+
+This dropped the unrelated proposal's confidence from 85% (always-maximal,
+regardless of content) to 16%, and the genuine paraphrase's from an
+identically-maximal 85% to a more proportionate 43% — both directionally
+correct. A regression test was added
+(`test_a_single_rejected_decision_does_not_swallow_unrelated_text`); the
+pre-existing "unrelated text" test never caught this bug because its
+fixture text had zero token overlap with the stored decision, never
+exercising the near-zero-but-nonzero BM25 score regime where the bug
+actually lived — a reminder that a test suite's own fixtures can share the
+synthetic-fixture trap's blind spots.
+
+## Consequences
+
+- `docs/BENCHMARK_PROTOCOL.md`'s "blocked on Consistency Engine existing"
+  language is now corrected — the engine exists, has a first real
+  measurement, and that measurement found and fixed a genuine bug, the same
+  pattern ADR-0009 and ADR-0016 established for other engines.
+- **Symbol-existence checking has a clean, confirmed 100% recall** on
+  invented function names in this pilot — the engine's strongest result so
+  far.
+- **The function-to-file relation blind spot is real and unfixed.** Closing
+  it would mean extending `claims.py`'s relation extraction to recognize
+  file targets and to tolerate explanatory text between the relation verb
+  and its arguments — a real change to the extractor's scope, deliberately
+  left for a future pilot/ADR rather than rushed into this one.
+- **Backtick-quoted local variable names are a genuine nuisance-false-
+  positive source** in real usage — anyone writing a normal technical
+  summary with `` `some_var` `` for readability will trigger a false
+  contradiction today. Worth a narrower fix (distinguishing "this claims a
+  codebase symbol" from "this is emphasis") in a future pass; not attempted
+  here to keep this pilot's scope to measurement plus the one clear,
+  high-confidence bug it found.
+- **The resurfacing fix is a real improvement, not a complete solution.**
+  BM25's IDF is inherently unstable with 1-2 documents; a small residual
+  false-positive risk remains for very small decision corpora. Stated
+  honestly rather than oversold, per this project's standing practice (T1).

experiments/consistency_pilot_1_hallucination_detection/README.md

@@ -0,0 +1,120 @@
+# First real measurement of the Consistency Engine
+
+`docs/BENCHMARK_PROTOCOL.md` said hallucination/contradiction detection was
+"additionally blocked on the Consistency Engine existing at all (Phase 3)."
+That sentence was stale: `src/verityai/consistency/` already exists, is
+tested (`tests/unit/test_consistency_check.py`), and was accepted in
+ADR-0007. What was actually missing was the same thing Context Engine
+lacked before ADR-0009: a measurement against real, not hand-authored,
+input. This experiment is that measurement.
+
+**Status: complete, and it found a real bug**, fixed in
+`src/verityai/consistency/check.py` (see `docs/adr/0018-consistency-engine-
+first-measurement.md`).
+
+## Part A — inducing real hallucinations, not writing them by hand
+
+Writing false claims by hand would measure the fixture, not the checker —
+the same synthetic-fixture trap `BENCHMARK_PROTOCOL.md` warns about
+elsewhere. The honest way to get genuine hallucinations: show a real agent
+only `billing/invoice.py` (from the pilot 5 fixture, reused here with a
+real code graph built via `verity graph build`) and ask it to describe the
+full call chain anyway, including functions it cannot see. This reliably
+produces a mix of true claims (what it could actually see) and invented
+ones (what it guessed about `apply_tax`/`apply_late_fee`'s internals) —
+five independent trials, five independently-generated transcripts.
+
+Ground truth (`ground_truth.md`) was written from the real source files
+*before* running `verity check` on any transcript, to avoid biasing the
+labels: `apply_tax` and `apply_late_fee` call **nothing** internally — they
+only read module-level dicts directly. Any claim that either function
+"calls" a helper or a file is false, whatever name is used.
+
+### Result
+
+98 claims checked across 5 trials, 23 flagged as contradictions:
+
+| Class | Count | Checker behavior |
+|---|---|---|
+| Invented helper-function names (`get_tax_rate`, `compute_overdue_penalty`, etc.) | 14 | **100% caught** — every single one flagged `FAIL` |
+| Backtick-quoted local variable names (`with_tax`, `days_overdue`) | 8 | Flagged `FAIL` as "no definition found" — **false positives**: these are real names in the real code, just not graph-indexed (the graph tracks functions/classes/files, not local variables) |
+| A genuine path inaccuracy (`invoice.py` instead of the real `billing/invoice.py`) | 1 | Correctly caught — a real, if minor, inaccuracy |
+| Fabricated relation claims ("`apply_tax` calls into `billing/tax_rates.py`", "`apply_late_fee` calls into `billing/policy.py`") | ~6 across trials | **0% caught — invisible to the checker entirely**, not even counted among the 98 |
+
+The last row is the important finding. Confirmed with isolated probes
+(not just inferred from the trial transcripts):
+
+```
+`apply_tax` calls `billing/tax_rates.py` to resolve the rate.
+```
+→ 0 contradictions. Decomposes into two independent, both-true existence
+checks (`apply_tax` exists, `billing/tax_rates.py` exists) — the relation
+extractor never fires when the "calls" target is a file rather than a
+function, regardless of how tightly the claim is phrased.
+
+```
+`apply_tax` calls `get_tax_rate` to resolve the rate.
+```
+→ 1 contradiction, correctly reported as a relation-level failure
+(`'apply_tax' exists, but 'get_tax_rate' was not found either`). So a
+symbol-to-symbol relation claim, phrased tightly, *is* checked correctly —
+the blind spot is specific to symbol-to-file relations and to relation
+claims phrased with explanatory text between the verb and its targets.
+
+## Part B — a real bug found in decision resurfacing
+
+A `REJECTED` decision was fabricated (`Apply the late fee using
+DEPRECATED_POLICY directly...`), then two independent proposals were
+checked: one a genuine paraphrase of the rejected idea (should trigger
+resurfacing), one a completely unrelated caching proposal (should not).
+
+**Both triggered resurfacing, at first.** The unrelated proposal "resembled"
+the on-file rejected decision because `check_decision_resurfacing`
+normalized each BM25 score against the *maximum score among the stored
+decisions*, not against an absolute scale. With only one or two decisions
+on record — a very common case, not an edge case — the single closest-of-
+the-available-candidates decision always normalizes to a perfect 1.0,
+regardless of whether it shares any real content with the checked text.
+Confirmed directly by adding a second, obviously unrelated rejected
+decision ("store passwords in plaintext") and observing *both* decisions
+get flagged against a caching proposal that resembles neither.
+
+**Fixed** in `src/verityai/consistency/check.py`: normalize against the
+checked text's own best-possible score (itself matched against itself),
+not against the in-corpus max. This dropped the unrelated proposal's
+confidence from 85% to 16% and the genuine paraphrase's from 85% (an
+overstated, always-maximal confidence) to a more proportionate 43% — both
+directionally correct, though a small residual false-positive risk remains
+for very small decision corpora due to BM25's own IDF instability at that
+scale (see the ADR's limitations section). A regression test
+(`tests/unit/test_consistency_check.py::test_a_single_rejected_decision_
+does_not_swallow_unrelated_text`) locks in the fix; the pre-existing
+"unrelated text" test never caught this because its fixture text had zero
+token overlap with the stored decision, never exercising the
+near-zero-but-nonzero score regime where the bug actually lived.
+
+## Files
+
+- `fixture_repo/` — the pilot 5 `billing/` fixture, reused as-is, with a
+  real code graph built on top (`.verity/graph.db`, gitignored).
+- `trial_1.txt` .. `trial_5.txt` — the five real, independently-generated
+  transcripts from Part A.
+- `ground_truth.md` — written before running any checker output.
+- `resurfacing_trial/` — the fabricated rejected decision and the two Part
+  B proposals.
+
+## Known limitations of this pilot, stated up front
+
+- **One fixture, one function, 5+2 trials.** Same caveat as every prior
+  pilot in this project — see `docs/BENCHMARK_PROTOCOL.md`.
+- **The relation-extraction blind spot (function-to-file relations) was
+  not fixed here.** It's a real, confirmed gap, but fixing the extractor's
+  regex to recognize file targets — and to tolerate explanatory text
+  between the relation verb and its arguments — is a larger change than
+  this pilot's scope; it's documented in ADR-0018 as a finding, not
+  patched.
+- **The resurfacing fix reduces but does not eliminate the small-corpus
+  false-positive risk.** BM25's IDF calculation is inherently unstable with
+  1-2 documents; a genuinely bulletproof fix would need a different
+  similarity approach for very small decision corpora, which is out of
+  scope here.

experiments/consistency_pilot_1_hallucination_detection/fixture_repo/billing/__init__.py

@@ -0,0 +1,10 @@
+"""Invoice total calculation."""
+
+from billing.late_fee import apply_late_fee
+from billing.tax import apply_tax
+
+
+def calculate_invoice(order: dict) -> float:
+    subtotal = order["subtotal"]
+    with_tax = apply_tax(subtotal, order["region"])
+    return apply_late_fee(with_tax, order["region"], order["days_overdue"])

experiments/consistency_pilot_1_hallucination_detection/fixture_repo/billing/invoice.py

@@ -0,0 +1,12 @@
+"""Late-fee calculation."""
+
+from billing import policy
+
+
+def apply_late_fee(subtotal: float, region: str, days_overdue: int) -> float:
+    region_policy = policy.DEPRECATED_POLICY.get(region)
+    if region_policy is None:
+        return subtotal
+    if days_overdue > region_policy["grace_days"]:
+        return subtotal * (1 + region_policy["fee_rate"])
+    return subtotal

experiments/consistency_pilot_1_hallucination_detection/fixture_repo/billing/late_fee.py

@@ -0,0 +1,15 @@
+"""Late-fee policy configuration."""
+
+ACTIVE_POLICY = {
+    "US": {"grace_days": 10, "fee_rate": 0.015},
+    "EU": {"grace_days": 14, "fee_rate": 0.010},
+    "UK": {"grace_days": 14, "fee_rate": 0.010},
+}
+
+# Superseded by ACTIVE_POLICY after the 2024 collections policy update;
+# nothing in the current late-fee pipeline should read this.
+DEPRECATED_POLICY = {
+    "US": {"grace_days": 30, "fee_rate": 0.020},
+    "EU": {"grace_days": 30, "fee_rate": 0.015},
+    "UK": {"grace_days": 30, "fee_rate": 0.015},
+}

experiments/consistency_pilot_1_hallucination_detection/fixture_repo/billing/policy.py

@@ -0,0 +1,8 @@
+"""Tax calculation. Healthy -- correctly reads the current rate table."""
+
+from billing import tax_rates
+
+
+def apply_tax(subtotal: float, region: str) -> float:
+    rate = tax_rates.REGION_RATES.get(region, 0.0)
+    return subtotal * (1 + rate)

experiments/consistency_pilot_1_hallucination_detection/fixture_repo/billing/tax.py

@@ -0,0 +1,15 @@
+"""Tax configuration."""
+
+REGION_RATES = {
+    "US": 0.07,
+    "EU": 0.20,
+    "UK": 0.20,
+}
+
+# Superseded by REGION_RATES after the 2024 tax reform; nothing in the
+# current tax pipeline should read this.
+LEGACY_REGION_RATES = {
+    "US": 0.05,
+    "EU": 0.19,
+    "UK": 0.20,
+}