Skip to content

security-guidance: commit and push reviews no longer hold a large diff in memory - #7140

Merged
mhegazy merged 3 commits into
mainfrom
mhegazy/sg-commit-diff-memory
Nov 18, 2026
Merged

mhegazy merged 3 commits into
mainfrom
mhegazy/sg-commit-diff-memory

Conversation

@mhegazy

@mhegazy mhegazy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Human: @mhegazy

What changes

The commit and push reviews read the whole git show -p / git diff into memory, decoded it and split it before looking at the number of files. On an import of 5,800 files and 4.6 M lines that took 6.4 GB, and the review was then skipped for its file count. They now read git's output as it arrives and keep only what the review uses:

  • The characters of each reviewed file that the prompt takes (DIFF_PER_FILE_BYTES, 80,000), and a count of the added lines after them. That count decides which files are reviewed when there are more than 30.
  • Past the cut-off of 300 files, only the file names. The skip still reports the full count.
  • For files that are not reviewed, the lines before their content.

Skipped commits and pushes also report diff_mb and diff_timeout, when not zero.

Fixes #7136.

What does not change

Which commits and files are reviewed, and what is sent. The end-of-turn review has its own reader, which is not touched. The combined diff of a merge's conflict resolutions is still read whole.

Limits

  • Added lines past a file's first 80,000 characters are counted up to 50,000.
  • Below the cut-off, 300 large files still cost about 135 MB, and more for text such as Chinese or Japanese (335 MB in one test).

Memory

Peak of the hook process, commit review, Linux:

Commit 2.0.20 this change
5,800 .py files, 4.6 M lines of 150 bytes, one emoji 6,350 MB 53 MB
5,800 .py files, 4.6 M lines of 80 bytes 1,608 MB 50 MB
300 .py files of 8,000 lines, an emoji in each (reviewed) 2,730 MB 135 MB
4 x 50 MB .json (reviewed) 981 MB 36 MB

Amend and push are the same. On Windows and macOS the import takes 46 to 63 MB. Half of the import on 2.0.20: 4.1 GB on macOS. SIGTERM during such a commit now ends the hook at once; on 2.0.20 it waits 1.3 to 1.5 s for the decode or split to return.

Tests

New: tests/test_commit_diff_memory.py, 117 tests. One existing test changes: test_wait_for_holder_stops_at_the_limit (from #7103) accepts a killed holder's lock released up to 5 s late, which Windows does on some machines, and checks that a file nobody holds is still no wait at all.

Runs are on this branch merged with main 1331242, against that main, unless a row says otherwise.

Scenario main This change Linux macOS Windows
Plugin suite Linux 2214 passed, macOS 2221; Windows x64 and ARM64 15 failed, 1989 passed Linux 2331 passed, macOS 2338; Windows x64 and ARM64 the same 15 failed, 2106 passed run run run, x64 and ARM64
The changed test, 6 times Linux not repeated; passes on the hosted machines, so the late release was not seen there passes run run run, x64 and ARM64
Hook memory on large commits (commit, amend, push, quiet commit) 0.3 to 4.3 GB on macOS and Windows (measured on 2.0.20's main), table above for Linux 30 to 135 MB run on the branch before the merge run run, x64 and ARM64
2,300 real commits of two repositories read whole and as here same files, first 80,000 characters and counts run, 800 of them on the merged tree not run not run
Requests sent for 15 large commits byte-identical to main run on the branch before the merge not run not run
22 scenarios: commit, push, end of turn, quiet commit, merge push, pattern, failed review, secret-shaped names, nested repository same as main in each run next row next row
Of those: commit, commit of staged work, push, end of turn, nothing changed, merge conflict push same as main run run run, x64 and ARM64
Very large commit and push with real Claude Code (5 cases) Linux peaks 791 to 7,054 MB (2.0.20) Linux peaks 54 to 89 MB, same reviews and findings run run; the three 5,800-file cases not on main there (7 GB machine) run, x64 and ARM64
Amend of a large commit with real Claude Code not run reviewed, finding delivered, 88 MB run on the branch before the merge not run not run
Regression bed Linux and macOS 21 of 21; Windows x64 8 pass, 8 skipped same run run run, x64
Hook killed while it reads the diff (SIGTERM, SIGKILL) nothing left behind nothing left behind run on the branch before the merge not run not run

On Windows x64, main logged no commit review for two of the five very large commits in this run and no finding for one of them; this change reviewed or skipped all five as expected. On Windows ARM64 the commit and push cases ran on the single-shot fallback review, as on main. The end-to-end runs use a scripted model.


Generated by Claude Code

The commit and push reviews captured, decoded and split the whole
`git show -p` / `git diff` before looking at the number of files. An
import of 5,800 files and 4.6 M lines took 6.4 GB, and the review was
then skipped for its file count.

Read git's output as it arrives and keep what the review uses: the
characters of each file the prompt takes, a count of the added lines
after them, and past the file cut-off only the file names.

Co-authored-by: mhegazy <mhegazy@anthropic.com>
claude and others added 2 commits November 18, 2026 00:11
…f-memory

Co-authored-by: mhegazy <mhegazy@anthropic.com>

# Conflicts:
#	plugins/security-guidance/hooks/security_reminder_hook.py
Windows can release a killed process's file lock a moment after the
process ended, so the wait that follows may see it held once more.

Co-authored-by: mhegazy <mhegazy@anthropic.com>
@mhegazy
mhegazy marked this pull request as ready for review November 18, 2026 01:01

@notitatall notitatall left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/stamp

@mhegazy
mhegazy merged commit e6143c9 into main Nov 18, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

security-guidance: commit-review hook buffers the whole git show -p before its file-count cut-off (multi-GB on large commits)

3 participants