[kensai] gitignore-aware stat-free file indexing - #7
Merged
Conversation
Port of the Go matcher-chain implementation. parsePattern compiles one .gitignore line via picomatch; Matcher chains per-directory pattern lists with last-match-wins semantics, child patterns checked before parent. Divergences from stock picomatch for gitignore fidelity: - [!abc] character classes normalized to [^abc] - dot:true -- globs match dotfiles - nobrace/noextglob -- braces and extglobs stay literal Consumed by the pathindex walker; integration follows in a separate change.
Index build on large repos took tens of seconds: every file was statted and ignored trees (node_modules, build output) were walked in full. Both costs removed: - Walker loads .gitignore per directory, chains it onto the parent matcher, and prunes ignored entries without descent; .git/info/exclude seeds the chain. Re-inclusion via ! inside a pruned directory is impossible, matching git. - Files are not statted during the walk. IndexEntryFile.size is null until lazily filled by RepoFs.readFile (buffer length) or a manifest stat. Symlink resolution stays. - PathIndex.add() inserts walk-pruned paths on demand (lstat-classified); used by the review layer to reconcile changed tracked-but-ignored files. - list_dir prints "(N bytes)" only when size is known. - readFile fallback for unindexed paths uses lstat and rejects non-regular files: a gitignored symlink pointing outside the repo could bypass path containment through the stat-following fallback. Walk of a 1.1M-file tree drops from 17.9s to 1.2s; the 172k-file dagor repo indexes in 1.0s.
The index is stat-free and gitignore-filtered, so buildManifest stats each changed file itself, filling the entry's lazy size, and inserts changed files missing from the index via index.add(). Without the reconciliation, tracked-but-ignored files (committed before the ignore pattern existed; 123 such files in dagor) silently dropped out of the manifest and stayed invisible to find_files/list_dir while still appearing in the diff under review. The diff is the source of truth, and git stays out of index construction.
IndexEntryFile.lineCount and maxLineLen were always 0: line metrics are computed on demand by countFileLines and never written back to the entry. Dead since line counting moved out of the index build (e8e980b).
Docs drifted from the May 27 read_file rewrite: repofs documented a streaming readFile(path, opts?) with line windowing options and a listDir method, neither of which exists. readFile returns raw bytes (windowing lives in the read_file tool); directory listing is served by the list_dir tool from index.dir(). Dependencies updated to match actual imports; lineiter's consumer is countFileLines, not the read tool.
On Windows picomatch auto-enables windows mode and converts backslashes in patterns to path separators. Gitignore \-escapes (escaped trailing space, literal backslash) compiled into patterns that never match anything. windows:false keeps backslashes as escape characters; patterns and matched paths are POSIX on all platforms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Index build (
session_start) on large repos took tens of seconds: the walker statted every file and descended into ignored trees (node_modules, build output) in full.Changes
src/repofs/git/gitignore/) -- pattern parser + hierarchical matcher chain ported from the Go reference. Full spec coverage: negation, dir-only, anchoring,**, char classes ([!abc]normalized for picomatch), dotfile matching, literal braces/extglobs..gitignorechained onto the parent matcher,.git/info/excludeseeds the chain. Ignored files omitted, ignored directories pruned without descent. Indexing stays a pure filesystem concern -- git is not involved.IndexEntryFile.sizeis null until lazily filled byRepoFs.readFile(buffer length) or a manifest stat. DeadlineCount/maxLineLenfields dropped.buildManifeststats changed files (filling lazy sizes) and inserts changed files missing from the gitignore-filtered index viaPathIndex.add(). Covers tracked-but-ignored files (123 in dagor), which otherwise silently dropped out of the manifest while present in the diff.readFilefallback for unindexed paths useslstatand rejects non-regular files; a gitignored symlink pointing outside the repo could bypass path containment through the stat-following fallback.Numbers (warm cache)