A recurring, expensive failure mode when an AI builds and checks its own work — worth understanding before you trust any "all tests pass."
What happens. An agent writes a tool, writes the tests for it, runs them green, and reports it working. Escalating the rigour — more cases, confidence intervals — does not correct the error; it makes the error more convincing. A detector can report 92% on a corpus it generated and 8.8% on real inputs, with the statistics arithmetically correct and epistemically worthless.
Why. The evaluation was a self-oracle: the same mind supplied the code, the test inputs, and the expected answers, so the suite could only ever confirm the author's own assumptions. The corpus contained only the failure modes the author could imagine — which is not the same set as the failure modes that exist. Knowing this principle does not prevent it; a competent essay on "generator ≠ critic" can be written the same morning the mistake is made.
The two halves — they need different antidotes:
| Half | The failure | The antidote |
|---|---|---|
| A — tests agree with the code | Assertions encode the same assumptions the implementation does, so the suite cannot fail. | Mutation testing — mutants come from a mechanical walker, so author bias can't shape which faults are proposed. |
| B — the corpus only holds what the author imagined | Even a perfect oracle is useless when every input was invented by the mind that wrote the code. | Externally-sourced inputs — a real labelled corpus, rival implementations, metamorphic relations. |
Mutation testing perturbs the code; it never invents an input. It tells you your assertions are weak — it can never tell you your corpus is fantasy. Fixing half A and feeling done is the same error one level down.
What this repo does about it. It treats independence as structural, not motivational: the
verification gate requires an artifact rather than a report; the strategy
cards differential-testing-vs-external-oracle
and independence-verification encode the antidotes; and
the standing rule is to frame verification adversarially — "this is broken, produce the evidence" —
never "please review."
The names to use: construct validity (the corpus measured "agreement with the author's model," not the capability), confirmation bias in testing (the mechanism), and the test-oracle problem (the structural gap — here, a self-oracle).