|
| 1 | +# Domain-ambiguity pilot: a fifth ceiling, for a genuinely new reason |
| 2 | + |
| 3 | +Per `docs/BENCHMARK_PROTOCOL.md`'s Family B procedure. Pilots 4-6 (ADR-0015, |
| 4 | +0016, 0017) all used bugs with **one answer derivable from the code alone**, |
| 5 | +and all three ceilinged on success — a capable model, given enough time, |
| 6 | +always found it. This pilot changes the shape of the difficulty entirely: a |
| 7 | +bug whose correct fix depends on a **business-rule ambiguity the code |
| 8 | +cannot resolve at all**, deliberately designed so the one visible test |
| 9 | +cannot distinguish the two plausible answers. |
| 10 | + |
| 11 | +**Status: complete, and it is a fifth ceiling — but for a new reason.** |
| 12 | +10/10 trials, both conditions, independently converged on the same |
| 13 | +(correct) interpretation of an ambiguity that genuinely wasn't in the code. |
| 14 | + |
| 15 | +## The design |
| 16 | + |
| 17 | +`fixture_repo/billing/late_fee.py`'s `apply_late_fee()` is a one-line |
| 18 | +no-op — trivial to spot, not the point. The real question: does the exact |
| 19 | +day a grace period ends (`days_overdue == grace_days`) still count as |
| 20 | +within grace, or is it already overdue? That's a policy choice |
| 21 | +(`days_overdue > grace_days` vs `days_overdue >= grace_days`), not |
| 22 | +something the surrounding code states anywhere. |
| 23 | + |
| 24 | +The one visible test (`days_overdue=20, grace_days=10`) is unambiguously |
| 25 | +overdue under either interpretation — it was deliberately chosen so both |
| 26 | +fixes pass it identically. A second, **hidden** test |
| 27 | +(`days_overdue == grace_days`, never shown to any trial) is scored |
| 28 | +independently by the harness afterward: the fabricated phase-A decision |
| 29 | +states the real policy is inclusive (fee only applies when `days_overdue > |
| 30 | +grace_days`, strictly), so the correct answer at the boundary is "no fee." |
| 31 | + |
| 32 | +This is structurally different from pilots 4-6: there, a careful-enough |
| 33 | +reading of the code always found the one right answer. Here, **no amount |
| 34 | +of reading the code finds it** — only the decision (recoverable via |
| 35 | +`verity handoff`) or an outside guess does. |
| 36 | + |
| 37 | +## Phase A, conditions, scoring |
| 38 | + |
| 39 | +Same shape as pilots 4-6 (`../lib/setup_phase_a.sh`): `naive` gets the bare |
| 40 | +task and no `.verity/`; `verity` gets a `.verity/` pre-loaded with the |
| 41 | +fabricated decision stating the inclusive-grace-period policy explicitly. |
| 42 | +5 live-agent trials per condition, Bash access, constrained not to touch |
| 43 | +`tests/`. |
| 44 | + |
| 45 | +Two metrics, both scored independently by the harness, never from the |
| 46 | +agent's own report: |
| 47 | +- `visible_pass`: does the shown test pass? (sanity check, expected ~10/10) |
| 48 | +- `boundary_correct`: does `calculate_invoice({"subtotal": 1000.0, |
| 49 | + "days_overdue": 10, "grace_days": 10})` return `1000.0` (no fee, the |
| 50 | + correct policy) rather than `1020.0`? |
| 51 | + |
| 52 | +## Result |
| 53 | + |
| 54 | +| Metric | naive | verity | Verdict | |
| 55 | +|---|---|---|---| |
| 56 | +| `visible_pass` (5 trials) | 5/5 | 5/5 | ceiling, as designed | |
| 57 | +| `boundary_correct` (5 trials) | 5/5 | 5/5 | `indistinguishable_from_noise` | |
| 58 | + |
| 59 | +Every one of the 5 `naive` trials independently wrote `days_overdue > |
| 60 | +grace_days` (strict) with no prompting toward the boundary question and no |
| 61 | +access to the decision — the same choice the `verity` trials made after |
| 62 | +reading it explicitly. None discussed the boundary case in their reasoning; |
| 63 | +`>` appears to be a strong default convention for "grace period" semantics |
| 64 | +regardless of whether the model was told the policy. |
| 65 | + |
| 66 | +**This is a genuinely different finding from pilots 4-6's ceilings.** There, |
| 67 | +success ceilinged because the answer was in the code and a careful agent |
| 68 | +always found it. Here, the answer was *not* in the code — and the ceiling |
| 69 | +happened anyway, because the model's default convention for this |
| 70 | +particular kind of naming (`grace_days`, "grace period") happens to align |
| 71 | +with the fabricated policy. That is a fact about this specific ambiguity |
| 72 | +and this model, not evidence the design failed: a different, less |
| 73 | +linguistically-loaded ambiguity (e.g., an arbitrary tie-breaking rule with |
| 74 | +no common convention) might show a real split. Reproduce: |
| 75 | + |
| 76 | +```bash |
| 77 | +./setup_phase_a.sh |
| 78 | +# run each of the 10 trials as a live agent per the design above |
| 79 | +verity noise-floor naive_results.json verity_results.json --metric boundary_correct |
| 80 | +``` |
| 81 | + |
| 82 | +## Known limitations of this pilot, stated up front |
| 83 | + |
| 84 | +- **The chosen ambiguity had a dominant linguistic prior.** "Grace period" |
| 85 | + strongly suggests an inclusive boundary in common usage and probably in |
| 86 | + training data (real billing systems, contract language). A genuinely |
| 87 | + unresolvable ambiguity needs a convention with no dominant default — e.g. |
| 88 | + a tie-breaking rule between two equally common but incompatible |
| 89 | + approaches, or a numeric edge case with no natural-language framing at |
| 90 | + all. |
| 91 | +- **This was the cheapest possible test of the design**, not proof the |
| 92 | + mechanism can never show an effect. A harder version of this same idea |
| 93 | + (an ambiguity without a linguistic tell) is the natural next pilot. |
| 94 | +- **One fixture, one ambiguity, 5 trials per condition.** Same caveat as |
| 95 | + every prior pilot — see `docs/BENCHMARK_PROTOCOL.md` before generalizing. |
0 commit comments