rule-taxonomy.md

Rule taxonomy

Every rule is classified on three independent axes: its effect (what a finding means for the person reading it), its evidence class (how far the rule has been validated) and its analysis scope (how much of the setup the rule must see). The axes are recorded on RuleMeta (effect, tier and scope) and are the single source of truth; the tables in rules-reference.md and the README are generated from them by scripts/gen_rules_reference.py.

Axis 0: effect

The effect is the one axis the commands select on. It answers "what should a reader do with this finding?" and is independent of severity.

Effect	Meaning	Who runs it
block	A decidable defect in the configuration: a referenced file is missing, a JSON key is duplicated, a frontmatter key is misspelt so the client ignores it, a secret is committed. Nothing to argue about.	harness-gate (gating tier), harness-autonomy (gating + provisional, as FAIL), harness-security (security category only)
policy	A decidable fact whose acceptability is a trust decision: permission prompts disabled, a wildcard Bash grant, a floating container image.	harness-autonomy (as REVIEW_REQUIRED unless a trusted policy file accepts it), harness-security
signal	A heuristic match: a text pattern, a taint flow, an AST or YARA signature. A claim for a reader, not a verdict.	harness-security only
advice	Quality and style: vague descriptions, redundant guidance, token budget.	harness-lint

Two invariants are tested: a signal or advice rule is always advisory tier, and a block or policy rule is gating or provisional. So the autonomy set is exactly "decidable and corpus-passed", by construction.

The effect is not the severity. A block rule can default to warning (hooks/valid-structure) and a signal rule to error (security/reverse-shell); severity says how loud, effect says what kind.

Axis 1: evidence class (tier)

A rule's evidence is either structural or heuristic.

• Structural rules are decidable from the files themselves. "This .mcp.json pins no version", "this referenced file does not exist on disk", "these two allow grants contradict a deny" are facts, not opinions. A structural rule can be promoted to a validated tier once the corpus study shows it is right in practice, or placed at gating when the condition is a filesystem or parse fact (missing file, duplicate JSON key, committed secret-named file, broken @import) rather than a precision estimate.
• Heuristic rules are judgments about prose: whether a description is vague, whether guidance is redundant, whether an instruction hedges. There is no ground truth in the file, only a signal, so heuristic rules are always advisory and never gate.

Structural rules carry one of three tiers:

• gating — safe to block a build on. harness-gate runs exactly the block rules of this set. Two ways in: corpus-validated at >=97% precision on >=50 re-derived findings plus a consequence review, or a decidable FILE/FILE_FS integrity check whose finding is a fact about the tree.
• provisional — zero false positives observed on the corpus, but on fewer than 50 findings, so precision is not yet established at the gating bar. Run them with harness-gate --include-provisional when you want an early signal.
• advisory — everything else, including every heuristic rule. Reported, never gated.

The gating and provisional sets are pinned by tests/test_rule_taxonomy.py, so a tier change is a deliberate, reviewed edit. The recommended and strict presets are derived from the same registry: every rule starts at RuleMeta.default_severity, then a small override map turns extras off or changes severity. A test requires every gating-tier ID to be present and not "off" in recommended, so those rules cannot silently drop out of default lint.

Axis 2: analysis scope

Scope records how much context a rule needs. It is what a per-file linter fundamentally cannot fake.

Scope	Needs	Example
FILE	One component's text or one settings/MCP file	mcp/unpinned-package reads a single .mcp.json server spec.
FILE_FS	That file plus the filesystem around it	claude-md/include-exists resolves an @import and checks the target exists on disk.
PAIRWISE	Two components compared against each other	content/duplicate-detection flags two skills that are near-copies.
SETUP	The whole component graph or an aggregate	cross/duplicate-skill-id needs every skill in the setup to know an id is ambiguous.

A per-file scanner sees one file at a time with no memory of the others, so it can decide FILE and FILE_FS rules but is structurally blind to PAIRWISE and SETUP classes: it can never notice that two skills duplicate each other, that two skills share an id, or that a credential flows from one component to a network sink in another, because those findings only exist in the relationships between files. Building the cross-component graph is the reason harness-eval exists.

Promotion criteria

A structural rule that needs corpus evidence is promoted to gating when the validation pipeline in the harness-eval-experiments repository shows both:

1. >=97% precision on >=50 re-derived findings — each candidate finding is independently re-derived at the repository's pinned commit by a check written without reference to the rule's implementation, and at least 50 such findings confirm at 97% or better.
2. A consequence review — the confirmed findings represent real problems (broken portability, a committed secret, a dead grant), not technically-true trivia.

Rules that clear the precision bar but on too few findings sit at provisional until enough evidence accumulates. Decidable integrity checks (missing file, duplicate JSON key, committed secret, broken import) may also sit at gating without waiting on that sample size. Rules whose evidence is linguistic never leave advisory.