Fix: stream the file cache as newline-delimited records to avoid RangeError on large collections

[terafin]

Problem

Cache (src/cache/cache.ts) serializes its entire contents into a single string:

• save: JSON.stringify(Object.fromEntries(this.keyValues))
• load: JSON.parse(Buffer.concat(chunks).toString('utf8'))

Once that string would exceed V8's maximum string length (require('node:buffer').constants.MAX_STRING_LENGTH, ~512 MiB), both throw:

RangeError: Invalid string length

This is hit on very large collections — I ran into it on a ~700,000-file collection, where the serialized cache is over 512 MiB. It is not an out-of-memory condition: raising --max-old-space-size doesn't help, because it's a string-length limit, not a heap limit. The only workaround today is --disable-cache, which throws away all cross-run cache reuse on exactly the large collections that benefit from it most. v5.4.0's gzip + streamed file I/O reduced the on-disk/heap footprint but still builds the one monolithic string in memory, so the crash remains.

Fix

Serialize the cache as newline-delimited JSON records — one [key, value] array per line — and stream them:

• save: a generator yields one gzipped JSON.stringify(entry) + "\n" at a time; no monolithic string is built.
• load: the gzipped stream is decoded incrementally with a StringDecoder, split on \n, and each line parsed individually.

No single string is ever built from the whole cache, so the size ceiling is now per-record rather than per-cache. save()'s existing write-validation (reload the temp file, compare entry counts) still holds because load and save use the same format.

Backward compatibility

A cache file written in the previous single-object format is not parseable as newline-delimited records; load() ignores it gracefully (an Array.isArray guard skips non-record lines) and the cache simply rebuilds on the next run. No migration or version bump needed.

Tests

Added to test/cache/cache.test.ts:

• round-trips values containing newlines, quotes, and backslashes (a newline-delimited format must not corrupt values that themselves contain newlines);
• ignores a legacy single-object-format cache file gracefully;
• asserts the on-disk format is one newline-delimited [key, value] record per entry, and round-trips back — this is the property that keeps any single string small.

I deliberately kept a literal ">512 MiB" reproduction out of the committed tests: reproducing the crash needs ~1 GiB of resident data and tens of seconds, which would be a heavy/slow CI addition. The format-invariant test proves the mechanism (one record per line, never one giant string) deterministically and cheaply.

npm run build, eslint, tsc --noEmit, and the full test/cache suite pass locally on Node 22.

[github-actions]

👋 Welcome

Thank you for your first pull request, @terafin! If you haven't yet, please familiarize yourself with Igir's contribution guidelines.

Some GitHub Actions CI workflows may not automatically run for you due to GitHub's security best practices, so a maintainer may need to manually approve the workflows to run. As a result, it is important to make sure tests pass locally before submitting a pull request to help ensure a fast review. Thank you!

Comment generated by the GitHub First Interaction workflow.