Skip to content

feat(redact): add a shannon entropy layer for unknown-vendor secrets - #44

Merged
theonly1me merged 1 commit into
mainfrom
feat/shannon-entropy-redaction
Sep 6, 2026
Merged

feat(redact): add a shannon entropy layer for unknown-vendor secrets#44
theonly1me merged 1 commit into
mainfrom
feat/shannon-entropy-redaction

Conversation

@theonly1me

Copy link
Copy Markdown
Owner

What changed

  • Added src/redact/entropy.ts with shannonEntropy, measuring bits per character over a token's own distribution.
  • Added a shannon-entropy rule that slices any token of 24 characters or more reaching 4.5 bits per character, keeping 7 leading characters.
  • Added slicedAboveEntropy to replace.ts, so the regex selects candidates and the redactor decides, with no change to RedactionRule.
  • Added 5 secret and 5 benign corpus fixtures, plus entropy.test.ts covering the function and the rule through redactSecrets.
  • Corrected the README so the entropy claim matches what the code does.

Why

README.md already advertised a Shannon entropy layer and none existed. Every other rule is a known-vendor pattern, so a credential in an in-house format passed through to the model untouched. Measured entropy is the only rule in the set that does not need to know the vendor in advance.

How to verify

bun run check

Ordering matters and is deliberate: the rule runs last. Placing it earlier lets it match a sub-token of a larger blob, redact only the tail, and block high-entropy-string from catching the rest. high-entropy-string is kept because it covers tokens with a mid-string = at 4.8 to 5.0 bits that this pattern splits below its floor.

The 24-character floor is arithmetic, not a tuned value. Entropy is bounded by log2(distinct characters), so 4.5 bits is unreachable below 23 characters. entropy.test.ts asserts that property rather than guarding it with a fixture.

Data handling

  • Any new capture source has an opt-in flag and a README entry in this PR
  • Everything that reaches the network passes redactSecrets
  • No raw capture in a log line, an error message, or a test fixture
  • A test exercises the real entry point, not just the redaction function

No new capture source, call site, or file reader. This adds one rule inside redactionRules, which is reached only through redactSecrets inside resolveRedacted. The single gate stays where it is.

@theonly1me
theonly1me merged commit ba1a52f into main Sep 6, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant