File indexer heuristics

[quadratech188]

I was testing a pull request when I noticed that a file in vicinae/src wasn't showing up in file search - after some investigation, it turns out that the /src pattern in .gitignore was ignoring it.

I understand that this is a hack (there's a comment saying so), but it made me wonder if handling .gitignore at this stage (while scanning) is the correct way. I would've been equally stumped if I couldn't search for the vicinae executable in /build.

Same goes for hidden files - Given how often I access my dotfiles, I think whether to include them should be an option when querying, not when scanning.

But we obviously need to ignore things like the .cache dir, and what should be ignored can be different for different people - I'd like some opinions on this.

[aurelleb]

These heuristics were implemented to keep scan performance reasonably fast (I was concerned about CPU spikes being problematic for some users).

• We could enable dotfile scanning by default rather than disabling it, with hardcoded exceptions for files like .cache. This would work well as a configurable preference.
• The gitignore pattern filtering could also be a preference. I'm uncertain whether it should be disabled by default, since developer systems typically contain many unwanted files that would otherwise get indexed (node_modules, build directories, etc.).

My overall thinking (feel free to challenge me on this) is that default indexing should prioritize speed over completeness. Users could then fine-tune coverage through preferences. I'd prefer to avoid subjecting new users to hour-long CPU spikes during their first experience.

[quadratech188]

Ok - I wan't really thinking about performance when I was writing this.

I did some benchmarking:

My NTFS data mount: (Through ntffs-3g and FUSE; I/O limited):

• With .gitignore - 380 seconds
• Without .gitignore - 95 seconds (probably because we don't backtrack through the path looking for .gitignore files)

My home dir: (BTRFS; CPU limited):

• With .gitignore - 77 seconds
• Without .gitignore - 67 seconds

It doesn't seem like there's much of a performance hit for ignoring .gitignore. Given I used Everything (Windows search utilify that indexes literally everything) for some time and was never bothered by 'unwanted files', I don't think we need this.
(Though the current implementation is very unoptimized - we should read .gitignore once when we iterate on it, not once for every file within)

I think you're right as for hidden files - but maybe make exceptions like .cache configurable too? (We can't possibly cover every cache directory name - node_modules could also belong here)

But FileSystemWalker seems to have a lot of opportunities for optimization - I'll try to work on that first.

[aurelleb]

Yeah I guess it really depends on what is on your system, me personally I have a lot of node_modules scattered everywhere and rust/c++/next.js build dirs everywhere as well so it can take quite a long time even on my pcie5 drive.

I will run a quick benchmark on mine to see what's the exact difference I'm getting.

Maybe we can just get rid of gitignore handling alltogether if this is not that big of an improvement.

Also yes, the gitignore logic is very much unoptimized, mainly because the directory crawling isn't the slow part on my system. But this totally changes if you are using I/O limited mounts.

[aurelleb]

@quadratech188 starting with v0.9.5 we don't use .gitignore by default (we may add a preference later)

[pavle99]

Just to confirm, .gitignore isn't taken into account and for now there is no way to enable it?