Description
Please tick this box to confirm you have reviewed the above.
- I have a different issue.
What version of ripgrep are you using?
ripgrep 14.1.0
features:-simd-accel,+pcre2
simd(compile):+SSE2,-SSSE3,-AVX2
simd(runtime):+SSE2,+SSSE3,+AVX2
PCRE2 10.42 is available (JIT is available)
How did you install ripgrep?
sudo pacman -S ripgrep (Arch Linux package manager)
What operating system are you using ripgrep on?
Arch Linux on WSL2
sh uname -a
Linux my-hostname 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 GNU/Linux
Describe your bug.
ripgrep process uses 7.1 Gibibytes of memory and is really slow (before getting killed by OS for allocating too much RAM),
when you run rg --files --ignore-file ~/.ultimate-gitignore, and .ultimate-gitignore being a big/huge .gitignore file, with 16332 lines (304KB), 9298 entries (or lines with empty lines and comments are removed)
this "Ultimate .gitignore" file is created by running cat * > .ultimate-gitignore in this directory github.com/toptal/gitignore/templates
What are the steps to reproduce the behavior?
If the corpus is too big and you cannot decrease its size, file the bug anyway and the ripgrep maintainers will help figure out next steps.
Nope, the corpus is not too big, it is only 2 files and a .git directory, I made sure to test it on a small corpus but my original use case was to use this command anywhere on disk, e.g. the $HOME directory
You can download these 2 files and .git directory by cloning this repo: wis/killall-for-Windows
and again, the command is: rg --files --ignore-file ~/.ultimate-gitignore
What is the actual behavior?
command:
rg --files --ignore-file ~/.ultimate-gitignore
output:
killall.c
zsh: killed rg --files --ignore-file ~/.gitignore
The command runs for 10 seconds and the process allocates 7.1 Gibibytes and has on average 50% CPU usage/utilization.
What is the expected behavior?
List all the files in the current directory, excluding the ones that match the patterns in the .ultimate-gitignore file
EDIT: I tried using fd, as Andrew recommends here:
You could also use a purpose driven tool specifically for searching by file name: https://github.com/sharkdp/fd/
but it also suffers from the same issue, the process allocates too much memory and gets killed by the OS.
I am starting to think this issue is fundamentally unfixable, given the nature of ripgrep's (and fd's) implementation,
I don't think you can "load" and compile this much patterns into memory and have this grantee by the regex crate:
This implementation uses finite automata and guarantees linear time matching on all inputs.
It's either ripgrep uses, or keeps using this fast regex implementation, which seems to me to trade off memory for speed,
or it's either ripgrep would be able to run this command, or a command that has this many patterns.
It's like:
- fast ripgrep (pattern matching)
- able to run this command
—pick one
EDIT2: I read the section on memory in the manpage, as recommended by Andrew in this comment here:
There are cases where ripgrep may use a lot of memory. They are documented in the man page. Please consult those to see if they match your use case.
...and I then ran the command above with the -j1 flag, as recommended in the manpage,
the command worked, it finished and exited sucessfully, but it was quite slow, even on a directory as small as the one mentioned above, with 2 files and 1 directory in it.
the command took 0.9 seconds to finish, on average, whereas rg --files takes 0.025 seconds (25 milliseconds) to finish, on average.