feature: Confidence Scoring & Tiered Detection

[addyCooks]

Description

Closes #95

Every Finding now carries a confidence (0.0-1.0) and a method naming the rule that
produced it, so an exact vendor key pattern (0.99, exact-pattern) is distinguishable from
a high-entropy guess (0.60, entropy) or a capitalised-word name (0.50, heuristic). The
scores reflect how the detectors already work this makes that implicit precision explicit
and filterable, so false positives no longer have to break automated workflows.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Refactoring

Testing Notes

Automated tests run? Yes 39 new tests, and the existing suite passes.

• Per-detector scores and methods, including both NameDetector modes.
• Threshold filtering for string and Message[] payloads, with stats reflecting the
  filtered set.
• Filter-before-collision ordering: a fabricated low-confidence Secret that outranks
  Email swallows the address unfiltered, but is dropped at a threshold so the email is
  detected instead.
• A detector reporting no confidence is scored at DEFAULT_CONFIDENCE.
• SecretDetector: a wider low-confidence overlap does not downgrade a vendor-prefixed key.
• Config schema validation (out-of-range and non-numeric minConfidence), flag-overrides-
  config precedence, and parseConfidence range rejection.
• End-to-end CLI: inspect display, inspect --min-confidence, scrub --min-confidence,
  hash agreement between the two, and rejection of out-of-range flag values.

Manual verification steps:

• Ran the exact command from the issue, scrub --min-confidence 0.8, on a payload mixing an
  OpenAI key, a high-entropy string, an email, a name and a bare URL: only the key (0.99) and
  email (0.95) were replaced; the 0.60 entropy string, 0.50 name and 0.70 bare URL were left
  alone. Without the flag, everything is replaced as before.
• Enumerated every built-in detector over a payload hitting all code paths: 26 findings,
  0 missing or out-of-range scores.
• Threshold sweep at 0.8 / 0.85 / 0.99 / 1 confirms the boundary is inclusive (a 0.80 finding
  is kept at 0.8, dropped at 0.85).
• No-change-by-default proved, not assumed: inspect --hash on this branch produces
  byte-identical output to upstream/main for two payloads covering every detector including
  the opt-ins.
• Rule-pack compatibility proved with a real package: built and installed a rule pack
  written against the original Finding interface (no confidence, no method). It loads,
  appears in rules list, scores 0.50 unspecified, and filters correctly at 0.5 and 0.8.
• Verified --min-confidence on watch --file --once, that init writes "minConfidence": 0,
  that config show round-trips a configured value, and that a malformed minConfidence is
  reported while scrubbing still falls back to 0.

Note on pnpm run check on Windows: test:types, test:lint, test:knip and
test:audit pass. test:format fails locally on 58 files including biome.json and
package.json, which this PR does not touch purely because this checkout has a CRLF
working tree while biome expects LF. The committed blobs are LF (git cat-file blob shows
0 CR bytes); extracting the commit to a clean LF tree and re-running biome ci reports
"Checked 59 files. No fixes applied." with zero errors, so CI sees a clean format check.

Similarly, 5 tests fail locally (cli/rules x3, core/rule-packs, storage) and a few CLI
suites hang, because this checkout path contains a space. They fail identically on a stashed
clean tree, so they are pre-existing and environment-specific; every other suite is green at
238 passed / 0 failed, and the new tests in the hanging suites were run individually via
ava --match.

Checklist

• If this was for an open issue, I was assigned to it
• I have self-reviewed my code.
• I have formatted, linted, and passed all tests (pnpm run check).
• I have added/updated documentation if necessary.
• I have added/updated tests if necessary.
• I have NOT bumped the version in package.json (maintainers will handle this).

[akramcodez]

@addyCooks Thanks for the PR! Could you please resolve the merge conflicts with main so we can get this reviewed?
Also, we recently migrated to Changesets please run pnpm changeset to generate a changeset for this new feature and commit the resulting file.

[addyCooks]

Thanks @akramcodez !
I’ve resolved the merge conflicts with main
and added the changeset as well.

[will-lamerton]

Nice work, and thanks for the thorough testing notes. I verified the compatibility claim myself: built both main and this branch, ran the README payload through inspect, and both produce the same hash. Default behaviour really is unchanged. Full suite passes here too (290 tests, tsc --noEmit clean).

Filter-before-collision is the right ordering, and the SecretDetector widest-span / strongest-evidence split is a genuinely subtle catch - good spot.

Four things before merge:

1. The new README hash is wrong. For the exact input in that block (including the trailing .), the real output is:

Hash: 7e5eea933db987e10e10e259ebcfea9d3250d8a68925fd9360f515e3a4bfbba9

I get that on both main and this branch. The old 41beda4a... was already stale and 66fcffd8... is a different wrong value. This one matters more than a normal doc typo because the surrounding prose is teaching people that the hash is deterministic and verifiable - anyone who copy-pastes to check will conclude the tool is broken. The Email_1 -> «Email_1» corrections in the same block are right.

2. Nothing tells the user what the threshold dropped.

$ echo "mail alice@example.com and call 555-123-4567" | prompt-scrub scrub --min-confidence 0.9
mail «Email_1» and call 555-123-4567
Scrubbed: 1 entity (1 Email)

The phone number was detected and then discarded, and the output is indistinguishable from "there was no phone number." For a redaction tool, silent under-redaction is the dangerous direction, and this is a flag aimed at automated workflows where nobody runs inspect first. Can we have the summary say so, e.g. Scrubbed: 1 entity (1 Email); 1 suppressed below --min-confidence 0.9 (1 Phone)? runDetectors already has both sets in hand.

3. parseConfidence accepts trailing garbage. --min-confidence 0.9zzz silently runs at 0.9, because Number.parseFloat stops at the first invalid char. That contradicts the function's own doc comment about not silently scrubbing more or less than the user asked for. Number(value) rejects the whole string and still handles 0, 1, .85, 9e-1. Worth adding '0.9zzz' to the existing rejection test.

4. The changeset overstates the guarantee. "existing output is unchanged unless the threshold is set" is true for scrub but not inspect, whose per-entity lines gained the confidence suffix unconditionally. Showing the score by default is the right call, but the changeset ships to consumers, so it should say the display changed and that --hash remains the scripting-stable surface.

Nits, take or leave:

• DEFAULT_CONFIDENCE is named in the rule-pack docs but isn't re-exported from src/index.ts, so a pack author has to hardcode 0.5.
• ARRAY_KEYS is now a hand-maintained list parallel to CONFIG_KEYS, so a future config key silently gets no validation. Net improvement over what was there, but a satisfies or a switch over CONFIG_KEYS would make drift a type error.
• The comment in name.ts is backwards: strict mode skips matches containing an allowlisted word, it doesn't remove anything from the allowlist.
• handleScrub passes minConfidence unconditionally while every neighbouring option uses conditional spread. Harmless, just inconsistent.

[addyCooks]

Thankyou @will-lamerton,
all four blocking items fixed, plus every nit. Also merged main in.

1. README hash. Reproduced rather than trusted: the block uses echo, which appends a newline 66fcffd8 is the value without it. Corrected to 7e5eea93….

2. Suppression reporting. runDetectors now returns dropped findings alongside kept ones, ScrubStats gains an optional suppressed, and the summary reads as you suggested:

Scrubbed: 1 entity (1 Email); 1 suppressed below --min-confidence 0.9 (1 Phone)

It fires when nothing survived too (Scrubbed: 0 entities; 1 suppressed…) the case where stdout is byte-identical to a clean prompt.

3. parseConfidence. Number() now, so 0.9zzz is rejected. 0, 1, .85, 9e-1 still parse; 0.9zzz, 0.5 0.6, NaN, Infinity added to the rejection test.

4. Changeset. Now states the inspect display changed unconditionally and that --hash is the scripting-stable surface.

Nits, all four done.

Four calls past what you asked, push back on any:

• inspect and watch report it too. Watch matters most: when a threshold drops everything the clipboard is untouched, so it would otherwise log nothing.
• A dropped finding whose span a surviving finding still redacts isn't counted detectors overlap constantly and a notice firing on ordinary overlap gets ignored. Tested both ways.
• handleInspect returns {findings, suppressed, minConfidence}, not a bare array churned the existing inspect tests.
• Collapsed runDetectors/runDetectorsDetailed into one function rather than leave a dead export.

Semgrep isn't this PR. main has 21 findings and the job runs --error, so every PR here shows it (non-blocking run concludes success, PR is UNSTABLE). Diffing result sets against main: the +3 was the stale ci.yml this branch still carried, and the code-tell.ts finding had only shifted line 19→23. Merging main brings it to 21, identical.

One real find on the way: 4 test files I touched were genuinely misformatted the CRLF working tree hides it, so I checked the LF blobs straight from git. Fixed.

Tests: 38 core, 32 e2e, 10 inspect, 22 config, 7 watch. CI unit tests pass in 38s.

Ready for another look.