Prevent requirement and test accretion from proof-of-wording incentives

[ncrmro]

Problem

The current specification and review workflow can reward more normative clauses, more test bodies, and more parameter cases without producing proportionally more evidence about product behavior.

Panopticon provides a concrete case study. A read-only audit found approximately 761 active requirements across 64 specification files. The audit's proposed consolidation retains the externally observable and security-sensitive contracts while reducing the normative corpus to roughly 400 requirements, primarily by merging artificial splits and cross-spec duplicates. Its test suite grew from 683 to 3,277 collected executions; an estimated 500–1,050 executions appear removable after the requirements are cleaned up.

These are advisory estimates, not deletion quotas. They demonstrate the incentive problem:

• one evidence obligation per normative item encourages one-test-per-MUST duplication;
• requiring rejected counterexamples for every conjunct and boundary term encourages Cartesian products over inputs that reach the same behavior;
• requirements about documentation, tests, CI, verdicts, or test helpers create prose pins and tests of test machinery;
• whole-file or shared-helper evidence hashes turn harmless edits into mass verdict invalidation;
• duplicated security rules across specifications produce duplicated evidence rather than one canonical invariant;
• requirements that prescribe an implementation mechanism obscure the observable outcome that actually matters.

This is not an argument against RFC 2119 language, adversarial review, security inventories, or agent-facing prompt contracts. It is an argument that the workflow currently rewards quantity and literal proof of wording more strongly than discriminating behavioral evidence.

Downstream case study and cleanup: Unsupervisedcom/panopticon-next#7

Reproducible evidence snapshots

These links intentionally identify commits rather than moving branches so a future reviewer or agent can inspect the same artifacts and reproduce or challenge the audit:

• Original Panopticon baseline at 7ebe39a
• Audited Panopticon-next snapshot at a485b8d
• 2119 implementation and reviewer-guidance snapshot at 652891f
• GitHub Spec Kit guidance inspected at 4a7341a

The counts and classifications in this issue describe those snapshots. Later changes SHOULD be compared against them rather than assuming the numbers still describe a moving default branch.

Prior art: GitHub Spec Kit

GitHub's Spec Kit offers useful requirement-authoring guidance that 2119 should preserve and sharpen:

• specify requires testable functional requirements and measurable, technology-agnostic success criteria.
• Its quality gate requires requirements to be unambiguous, scoped, connected to acceptance scenarios, and free of implementation details (checklist construction).
• Its core abstraction rule is to specify what users need and why, not how to implement it.
• Its requirements-checklist workflow evaluates completeness, clarity, consistency, measurability, and scenario coverage rather than testing the implementation (quality dimensions).
• Crucially for accretion, it tells checklist authors to merge near-duplicates and consolidate more than five low-impact edge cases.

That guidance is better than treating every normative phrase as an independent proof obligation, but it is not sufficient by itself. “Each requirement must be testable” can still be misread as “each requirement must have its own test,” while broad prompts to identify all edge cases can still reward speculative negative-space enumeration. The symmetric review probe proposed below adds the missing discriminator: evidence must fail when behavior is removed and survive legitimate changes that preserve meaning.

Distinguish three concerns

1. Keyword misuse: MUST is used for editorial choices, exact prose, repository layout, test construction, or implementation mechanisms that are not interoperability or harm-prevention contracts.
2. Workflow incentives: per-item evidence, exhaustive boundary-term counterexamples, and broad evidence hashes make requirement and test accretion a rational way to obtain a pass.
3. Repository-specific cleanup: repositories still own their duplicated requirements and low-value tests. The tool should change the incentives, not automatically delete evidence.

Proposed direction

1. Make the counterexample obligation symmetric

The reviewer SHOULD name:

• one concrete implementation change that violates the requirement and that the evidence would detect; and
• one legitimate change—such as paraphrasing, renaming, reformatting, adding a sibling item, or reorganizing files—that does not violate the requirement and therefore MUST NOT fail the evidence.

A test that breaks only because wording or organization changed is a tripwire, not behavioral evidence. This generalizes the falsifiability work in #30 without requiring a counterexample for every word in a requirement.

2. Count shared evidence, not test bodies

• One test MUST be allowed to cover multiple requirement IDs.
• Reviewers MUST NOT request a new test when existing evidence already rejects the relevant counterexample; they SHOULD request an additional annotation or explicit cross-reference.
• A requirement MUST NOT receive credit merely because it has a uniquely named test.

3. Separate product requirements from verification notes

Normative product requirements SHOULD describe observable behavior or a genuine delivered contract.

Test strategy, CI commands, minimum subject counts, review procedure, migration bookkeeping, and tool-version adoption SHOULD live in verification or maintenance notes unless they are themselves an externally supported interface.

Agent-facing text MAY be normative when it is delivered through a product surface and materially controls agent behavior. The contract SHOULD state the instruction or outcome that must survive; it SHOULD NOT pin an entire paragraph, heading order, or unrelated prose.

4. Narrow verdict invalidation

Verdict identity SHOULD be scoped narrowly enough that harmless fixture, helper, documentation, rename, or formatting changes do not invalidate unrelated evidence. This complements #27. The design MUST preserve detection of shared mocks or helpers that can genuinely neutralize evidence.

5. Use judgment rather than another metric gate

Hard limits on executions per requirement, prose-test ratios, branch counts, or matrix sizes would create another compliance game. The tool MAY report these as review prompts or trends, but they SHOULD NOT become pass/fail proxies for test value.

Reviewer questions

For each requirement/evidence relationship, the review packet SHOULD ask:

1. What public or delivered surface exposes the required behavior?
2. What plausible implementation change would violate it, and would this evidence fail?
3. What legitimate non-violating change would leave the evidence green?
4. Does existing evidence already cover the same counterexample?
5. Do parameter values exercise meaningfully different behavior, or merely different spellings of the same branch?
6. Is the requirement an outcome, or is it prescribing documentation, tests, CI, or implementation structure?

Anti-patterns to call out explicitly

• exact equality or substring assertions against specifications, general documentation, comments, or docstrings;
• repository-file, generated-string, or test-module digest pins used as behavioral evidence;
• frozen inventory literals where the inventory could be derived from the running product;
• Cartesian products whose dimensions do not interact in production behavior;
• new test files named for review rounds or gaps rather than product behavior;
• unit tests for test-only helpers created solely to satisfy the evidence gate;
• a skipped end-to-end test surrounded by always-running tests of its scaffolding;
• duplicated tests attached to split or renamed requirements.

Important exceptions

• Derived route, command, plugin, or harness inventories are valuable security evidence when zero subjects fail loudly and the inventory comes from the real product. This overlaps Scope inflation: a verdict can claim a plural subject from single-member evidence #17.
• Distinct malformed data shapes can justify negative-space tables.
• Agent-facing prompt text is behavior when tested through the actual rendered/delivered surface.
• Snapshot or golden tests remain appropriate when their supported update path is explicit and inexpensive.
• Reviewer-selected targeted mutation remains useful evidence, as proposed in Reviewers should break the code, not just read it: add a mutation step to judgment reviews #26.
• Skip-gated evidence still needs the visibility proposed in Skip-gated evidence is invisible to the gate and to reviewers #29.

Acceptance criteria

• Review instructions MUST include both a violating-change probe and a legitimate-change survival probe.
• Review instructions MUST permit one evidence body to cover multiple requirements and MUST discourage duplicate bodies added only for traceability.
• Author guidance MUST distinguish behavioral requirements, delivered-text contracts, and non-normative verification notes.
• The reviewer MUST reject evidence whose only value is pinning irrelevant wording, layout, hashes, or implementation organization.
• The reviewer MUST consider whether large parameter sets exercise distinct behavior rather than demand exhaustive cases from universal wording alone.
• Verdict invalidation scope MUST be documented, including the tradeoff between helper integrity and unrelated hash churn.
• Existing calibration examples MUST cover both false rigor and legitimate exceptions: prose tripwire, redundant matrix, derived security inventory, delivered agent instruction, and distinct malformed shapes.
• Rollout MUST begin as reviewer-guidance changes and observational reporting; it MUST NOT automatically delete or invalidate repository tests.

Evaluate without creating another target to game

Observe these as trends, not gates:

• verdict invalidations after changes that did not alter requirement meaning;
• review rounds per requirement;
• tests added versus existing tests cross-referenced when specifications change;
• time required for documentation-only changes to reach green;
• mutation detection of full matrices versus representative subsets;
• requirements covered only by tests skipped in default CI.

Related upstream work

• Scope inflation: a verdict can claim a plural subject from single-member evidence #17 — scope inflation and real-artifact enumeration
• Reviewers should break the code, not just read it: add a mutation step to judgment reviews #26 — targeted reviewer mutation
• Review cost is requirements × rounds: shared preludes mass-stale verdicts, adjacent annotations produce empty evidence blocks #27 — review cost and broad evidence hashes
• Skip-gated evidence is invisible to the gate and to reviewers #29 — skipped evidence visibility
• Add a falsifiability probe to the review packet #30 — falsifiability probes

[luce-unsup]

Reference check complete — everything I can inspect holds up. All five related issues exist and this issue's one-line characterizations match them (and the #30 relationship is accurate: this adds the symmetric legitimate-change survival probe to #30's violating-change probe). panopticon-next#7 exists, cross-links back, and every shared number agrees (64 spec files, ~761 requirements, ~400 target, 683 → 3,277 executions, 500–1,050 removable). All four pinned commit SHAs resolve, and the Spec Kit anchors land on the described content at 4a7341a — including checklist.md's "merge near-duplicates" / ">5 low-impact edge cases" consolidation rule.

Two notes and one nit on the text itself:

1. Antecedent bug in the case-study paragraph. "Panopticon provides a concrete case study … Its test suite grew from 683 to 3,277 collected executions" — per panopticon-next#7, 683 is the original Panopticon suite and 3,277 is the audited panopticon-next snapshot. As written, "its" attaches to Panopticon and reads as one repo's suite quintupling, when the comparison is cross-repo (rewrite vs. original). Naming panopticon-next explicitly would keep the accretion claim accurate.

2. Acceptance criterion 4 could eat the exceptions. "The reviewer MUST reject evidence whose only value is pinning irrelevant wording, layout, hashes, or implementation organization" is in tension with the Important-exceptions section, which blesses derived inventories and delivered agent-facing text. The word "irrelevant" plus criterion 7's calibration examples mostly cover it, but the criteria never state the carve-out — a literal implementer of the review instructions could reject a derived-inventory digest the exceptions endorse. One explicit clause ("…unless the pinned text is itself the delivered product surface or a zero-subject-fails-loudly derived inventory") or a pointer to the exceptions would close it.

3. Modal nit: criterion 5's "reviewer MUST consider whether large parameter sets exercise distinct behavior" — "MUST consider" is a free action, so it functions as the kind of compliance checkbox §5 argues against. The reviewer-questions list (Q5) already carries the substance; a guidance-level framing would be more self-consistent.

The pinned-SHA approach for the evidence snapshots is exactly right — reproducible and challengeable rather than pointing at moving branches.