perf(local): only re-read the files that changed (#411)

[TheZupZup]

Closes #411. Part of #376.

Second of four PRs on Linux local-library reliability. Independent of the others, based on latest main.

What changes

A local scan spends nearly all of its time opening files and parsing their metadata blocks. On a routine rescan almost none of those files changed, and it parsed every one of them anyway.

Now each candidate file is stat-ed for its size and last-modified time, which is one syscall. If the catalog already holds that path with an identical stamp, the stored track is reused verbatim and the file is never opened. Only new files and changed files get parsed.

Numbers

test/benchmarks/local_incremental_scan_bench.dart, on a synthetic tree with a fixed 200 microsecond per-file parse cost (this machine, so treat these as shape rather than truth):

files	full scan	rescan	parsed
2,000	635 ms	55 ms	2000 → 0
20,000	4,803 ms	151 ms	20000 → 0
100,000	23,327 ms	655 ms	100000 → 0

The parse cost is pinned rather than measured on purpose: a real tag read is dominated by I/O that varies by an order of magnitude between an SSD and a spinning USB drive, and by page cache state, so pinning it is what makes the two columns comparable. 200 microseconds is well under a cold FLAC tag read, so the saving shown is a floor.

Files

• lib/core/models/local_file_stamp.dart is the stamp (size + mtime) and the StampedTrack pairing. It documents what size and mtime miss, because that is what the full-rescan path exists for.
• lib/core/sources/local/local_file_stat.dart is the stat seam, beside the scanner and metadata-reader seams and for the same reason. The unsupported implementation answers nothing, so every file is parsed: that is what Android gets (its local library comes from the content resolver, so there is nothing to stat), and what any caller that has not opted in gets.
• lib/core/repositories/stamped_catalog_writer.dart is the optional repository capability, in the same shape as SourceCatalogReader and IncrementalCatalogWriter. Falling back to a plain upsert is always correct and only slower, which is the property that lets it be optional.
• LocalMusicSource does the comparison. LocalLibraryScanner carries stamps through the merge, including for retained tracks: an offline folder keeps its stamps, so plugging the drive back in does not re-parse it.

The schema migration, and why

Schema v5 adds two nullable columns to tracks: file_size_bytes and file_modified_at_ms.

The issue asks not to migrate without clear value, so here is the case. The stamp has to live somewhere that cannot disagree with the catalog. On the row, it is one write in one transaction, and there is no window in which a stamp says "already parsed" for a track the catalog does not actually have. A sidecar store would avoid the migration and buy exactly that hazard: an interrupted or partially-restored index would make a later scan skip files it never indexed, and the symptom would be a silently short library.

It is purely additive. Both columns are nullable with no default, so SQLite's ALTER TABLE … ADD COLUMN rewrites no rows: existing rows keep every value and read back with null stamps, and a null stamp means "parse this file", i.e. exactly the pre-v5 behaviour. Nothing to backfill, and nothing lost if the upgrade is interrupted. Tests cover v4 → v5, v2 → v5 in one go, and a fresh install.

Recovery

scanFolders(full: true) ignores every stamp and rebuilds the slice from the files themselves. It goes through the same code path (it hands the scan an empty index rather than taking a second route) so the two cannot drift apart. Forgetting the source and re-selecting the folders does the same thing, since there is then nothing indexed to compare against. A full rescan still records fresh stamps, so recovery does not cost a full parse on the next scan too.

Tests

They count parser calls, not results. Asserting that two scans produce the same catalog proves nothing about whether the second one re-read every file, which is the entire point: the catalog was already correct before this change, it was just expensive to arrive at.

• test/core/sources/local/local_incremental_scan_test.dart: a second scan of an unchanged library reads zero tags and stats each file once; a moved mtime, a changed size with an unchanged mtime, a new file, and a file that cannot be stat-ed each re-parse and nothing else does; a full rescan re-parses everything and still records fresh stamps; no stat reader means the old behaviour.
• test/features/library/incremental_library_scan_test.dart: the same through the real Drift repository over in-memory SQLite, because a stamp that does not survive the schema is not a stamp. Covers multiple roots, a changed file in one root not re-parsing the other, and a drive going offline and coming back without re-parsing.
• test/data/database/linthra_database_migration_test.dart: the v5 migration cases above.

Real-device / real-filesystem limits

Documented in docs/local-music.md:

• Size and mtime miss a write that restores both. A backup tool putting an old file back over a new one does it, and so does touch -r after an in-place edit. Hashing would catch those, at the cost of reading every byte of the library on every scan, which is the thing being avoided. The full rescan is the answer.
• mtime resolution is the filesystem's. ext4 and btrfs give nanoseconds, but this stores milliseconds (what SQLite holds and what FileStat reports portably), so two writes to one file inside the same millisecond that leave the size unchanged look identical. Vanishingly unlikely for audio files, and the full rescan covers it.
• A file the walk listed but the stat cannot answer for (a race with a delete, a permission change, a mount that went away mid-scan) is always parsed, never reused. Unknown never means skip.
• Network mounts report mtime from the server, so a clock skew there shows up as a re-parse, not as a missed change. The safe direction.
• Android is unaffected and scans exactly as it always did.

Checks

dart format, flutter analyze (clean), flutter test: 52 new tests, 4933 passing. Drift output regenerated with dart run build_runner build --delete-conflicting-outputs.

Two existing suites needed a real fix rather than a nudge. The deferred-scanner concurrency tests assumed the filesystem walk had already started the moment a scan future was created; that stopped being true once the scan reads the indexed slice first, so they now wait for the walk to genuinely start before superseding it, which is what they meant to assert. And the Library screen widget test now stubs the stat seam for the same reason it already stubs the tag reader, with the same comment: real filesystem I/O on fictional paths never completes against pumpAndSettle's fake clock.

Relationship to the other three

• [Linux] Handle moved and deleted local tracks #410 (moved/deleted tracks) is independent and can be reviewed in either order. They overlap only in library_controller.dart, where both want the previous slice available on every scan, and in two concurrency tests that both need the same restructure for the same reason. Expect a small conflict there, whichever lands second.
• [Linux] Add filesystem library watching #409 (filesystem watching) builds on this PR: watcher events have to drive an incremental update rather than a full rescan, which is the semantics this adds. Its branch will declare the dependency.
• [Linux] Add memory-usage regression check for large libraries #463 (memory harness) is tooling and touches no production behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lq4ZJtwtdVCMBuZLjFbyfV

---

Generated by Claude Code