Skip to content

security-guidance: count changed files that no Stop review saw - #7168

Draft
mhegazy wants to merge 1 commit into
mainfrom
mhegazy/sg-count-dropped-changes
Draft

mhegazy wants to merge 1 commit into
mainfrom
mhegazy/sg-count-dropped-changes

Conversation

@mhegazy

@mhegazy mhegazy commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Measure only. What is reviewed does not change.

A prompt replaces the baseline of the Stop review unless an edit tool recorded a path. Files changed in other ways since the last Stop review (commands Claude ran, the user, a pull) can then be in no Stop review. This counts how often.

What is sent

changed_files on a Stop row, only when it is not 0: 1000 * kind + files (reviewable files, at most 999). 10 is added to the kind when HEAD moved as well. -1 when git could not say.

Kind Meaning
1 no Stop since the last prompt: an interrupt, an API error, a message sent mid-turn
2 a turn that a finding started, whose Stop reviews nothing
3 the last Stop ran with an agent at work in the background
4 this row's own review failed and no edit tool recorded a path

No path or name is sent or kept.

What it cannot say

  • It is an upper bound: changes by Claude's commands, by the user and by a pull are counted alike.
  • Commands left running in the background are not counted, nor a session that ends before its next Stop.
  • For kinds 2 and 3, files deleted after the Stop are not counted.
  • Kind 4 is counted at once. A Stop before the next prompt still reviews those files.
  • A row with more than 20 keys loses this key before the others.

Cost

Nothing on a prompt after a turn that a Stop ended. Otherwise one git diff --name-only between two commits, at most 1 s in all: 135 to 139 ms for the prompt hook in a small repository, 200 to 204 ms with 40,000 files (Linux, medians). No extra save of the session state.

Tests

Scenario main this PR Linux macOS Windows x64 Windows ARM64
Real sessions: interrupt, message mid-turn, background agent, finding turn no key counted, the expected kind run run run run
Real sessions: API error, background agent with a prompt in between no key counted, the expected kind run not run not run not run
Real sessions: plain turns, edit tools in the same turns no key no key run run run run
Real sessions: foreground agent, queued message no key no key run not run not run not run
tests/test_dropped_changes_metric.py 39 of 39 run run run run
Plugin suite, failures not on main 0 run, 2,370 passed run, 2,319 passed run, the same 16 fail on main run, the same 16
Regression scenarios (gateway, managed MCP, commit only) 21 of 21 21 of 21 run not run not run not run

The sessions use a scripted model. No version bump.


Generated by Claude Code

A prompt replaces the review baseline unless an edit tool recorded a path, so files
changed in other ways since the last Stop review can be in no review. Count them on
the next Stop row (changed_files). What is reviewed does not change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants