red-handed

Your agent said the tests pass. This checks that a test actually ran.

_한국어

One command:

npx @jinhyuk9714/red-handed@latest

On my machine it read 601 sessions. My agent said "tests pass" 202 times. 10 of those times, there was no run it could read. What does yours say?

It reads the Claude Code session logs already on your disk, plus your git state. No account, no API key, no model call. Nothing leaves your machine. (npx itself fetches the package from the npm registry, the way it does for anything else.)

The run that was thrown away

There is a second, quieter way a run goes missing. To save context, agents pipe test output through a filter:

npx vitest run 2>&1 | tail -5

tail -5 keeps the last five lines, and Tests 33 passed is often above them. The run happened. The record kept nothing. On my machine 42% of test runs ended like that. red-handed receipts counts yours.

You can stop it with one command:

npx @jinhyuk9714/red-handed@latest fix

That installs a small hook. When the agent is about to run a test command that ends in a result-eating pipe, the hook removes the pipe, so the result reaches the record. Test output is a few hundred characters at the median, so keeping it costs almost nothing. It never touches permissions, only the command text, and only for test commands. unfix removes it.

It also audits, if you ask

Behind the count is the original tool: nine checks that line the session log up against your git state and report where what the agent said and what it did don't match, with timestamps and quotes.

When I wrote this up I had 249 sessions. The audit confirmed nothing, and flagged seven claims whose verification left no trace anything could read. The whole story.

Install as a plugin

Two lines, and after that there is nothing to remember:

claude plugin marketplace add sjh9714/red-handed
claude plugin install red-handed@red-handed

Installing turns on two hooks. One removes the result-eating pipe from a test command just before it runs. One audits the session against git the moment it ends, and says something only when it caught something. You also get /red-handed:audit and /red-handed:history for asking Claude directly. Uninstalling the plugin turns both hooks off again.

Not using the plugin? npx @jinhyuk9714/red-handed@latest fix and install-hook set the same two hooks up by hand.

On Codex, Cursor, Gemini CLI and other agents, the skills install with the skills CLI:

npx skills add sjh9714/red-handed

The nine checks

what it catches	id
the suite got smaller — a test that no longer runs cannot fail	test-census
said tests pass — the last run failed	claim-vs-fail
said tests pass — none ran, not even in a subagent	claim-no-run
rewrote the expected value to match the bug, code untouched	hardcoded-expected
replaced the assertion with one that cannot fail, or commented it out	assertion-weakening
switched a test off (.skip, .only, xit, @pytest.mark.skip)	skip-only
hook rejected the commit → committed again with hooks off	no-verify
turned a check off (strict: false, CI test step deleted, suppression on a failing line)	config-disable
wrapped a fresh error in a catch that does nothing	error-swallowing

Every finding carries the timestamp and the quoted line it came from, so you can open the transcript yourself and disagree.

No model is called. The whole thing is deterministic: the same session gives the same answer every time. Claims are matched in English, Korean, Japanese and Chinese; reports come in English and Korean (--lang ko).

What it does not do

It cannot tell you whether the code is right. It tells you when the agent's own record does not support what the agent said, which is a much smaller claim.

Known blind spots, up front:

• Claim sentences are matched in English, Korean, Japanese and Chinese. The report itself is written in English or Korean; a Japanese or Chinese session is read correctly but reported in English, because a translation nobody checked is worse than none. A session in an unlisted language produces fewer findings, never wrong ones. The patterns are a data file (src/claims/patterns.ts) if you want to add yours.
• Verification it cannot read is treated as verification it did not see. If your tests run through a script whose output this cannot parse, even a true claim only reaches SUSPICIOUS.
• Browser tests, manual checks and anything else without machine-readable output are invisible to it.
• --git-only mode has no transcript to read, so nothing it reports is ever more than a suspicion.

CAUGHT and SUSPICIOUS

CAUGHT means two things were both true: the session shows the agent doing it, and the code still shows it now. If the agent later undid the change, there is nothing to accuse it of, and the finding disappears.

SUSPICIOUS means the pattern is there but the motive is not established. An empty catch block is sometimes exactly right. A test skipped on purpose is sometimes exactly right.

The tool is tuned to miss things rather than to accuse wrongly, and it is worth saying what that costs. An adversarial review of this code found five separate ways it could print CAUGHT at honest work — a test run that timed out read as a failure, a runner it did not know (rspec, phpunit, tox) read as no run at all, a re-run under a different launcher it could not see, a requested behaviour change read as rewriting the answer, and a backgrounded command counted as a pass. All five are fixed and each has a permanent regression test in test/regression/false-accusations.test.ts. If you find a sixth, that is the bug report I most want.

Running it on my own history is where the sixth one turned up. A scoped pytest tests/test_sync_cli.py was followed by edits to a GitHub workflow and two .tsx files in a separate web app, and the tool called the claim stale. A Python test cannot import a .tsx file and a workflow is not what just ran, so nothing about that claim had gone stale. It now only counts a change the runner could actually have loaded.

Usage

red-handed                        how often "tests pass" had a real run behind it
red-handed audit                  audit the most recent session for this directory
red-handed audit --all            audit every session for this directory
red-handed audit --session <id>   audit one specific session
red-handed audit --git-only       audit the diff instead, when there is no transcript
red-handed stats                  every unbacked claim on this machine, quoted
red-handed fix                    stop the agent cutting test output out of the record
red-handed unfix                  undo that
red-handed receipts               count test runs whose result was thrown away
red-handed demo                   see every check fire, on a made-up session
red-handed install-hook           audit automatically when a session ends

Useful options: --json and --md for machine-readable output, --lang ko for Korean, --fail-on caught|suspicious|never for CI, --detectors a,b to run a subset, --no-cache to ignore the cache.

stats keeps a cache at ~/.red-handed/cache.json (mode 0600, relocate it with RED_HANDED_HOME) so repeat runs are instant. It holds excerpts from your sessions and never leaves the machine. Credentials are masked out of every quoted excerpt before it is written or printed.

Exit codes: 0 nothing found, 1 findings at or above --fail-on, 2 wrong usage.

In CI

- run: npx @jinhyuk9714/red-handed@latest --git-only --fail-on caught

Without a transcript this only reads the diff, so treat it as a smoke alarm rather than a verdict.

Forget it is there

npx @jinhyuk9714/red-handed@latest install-hook

After this, every Claude Code session is audited the moment it ends. When a session is clean — which is most of them — you see nothing. When something was caught, a one-line warning appears right in Claude Code:

red-handed: 1 finding(s) caught this session — run npx red-handed to see them

The audit takes about a tenth of a second and never blocks the session, whatever happens. Existing hooks are left alone and the old settings are copied to settings.json.red-handed-backup first. uninstall-hook removes it.

Requirements

Node 20 or newer. Claude Code session logs are read from ~/.claude/projects. Test output is parsed for vitest, jest, mocha, pytest, rspec, minitest, phpunit, python unittest, go test, cargo test and node --test. Runs are also recognised for tox, ctest and friends, plus make test, ./gradlew test, ./mvnw verify, bundle exec, and project scripts named like tests. Recognised means a claim about them stays honest, even where the output format is not parsed yet.

How it was built

Written with Claude Code. I wrote the spec and the false-positive rules, reviewed every detector, and then had the finished thing pulled apart by an adversarial review that reproduced each defect against the built binary before I believed it.

That review is where most of the guards came from. It also caught the tool doing the exact thing it exists to detect: the 0 CAUGHT release gate I had written into this README held because my own sessions are all JavaScript and TypeScript, so the code path that mis-read a timeout as a failure had never once run. The gate passed by luck and reported success — the exact move this tool exists to catch.

Related work

This is not the first tool to try this. I checked before assuming otherwise.

• agent-receipts reads the same transcripts for the same purpose, and got there first. It is Python, English-only, and does not check whether the code it accuses still looks that way today.
• claude-tap intercepts agent API traffic while it happens. This reads the transcript afterwards.
• Session viewers such as claude-code-session-viewer let you browse transcripts by hand.
• vibe-kanban orchestrates agents; it does not audit them.

What is different here is the tiering — a CAUGHT needs the change to still be in your working tree — and how hard that was to get right. If your tool belongs on this list, open a pull request.

License

MIT