Advice for allocations when handling very large files

[T145]

I'm aware that ripgrep has set memory limits that can cause problems, and that has been encountered while trying to perform the following operation:

rg -NFxvf source.txt target.txt

Where the source is ~8MB and the target is ~168MB. What should be the memory allocations I use in order to handle this workload?

[BurntSushi]

Umm... I don't know. And literally can't know without seeing source.txt. Why not try some limits and experiment with it? Even given my knowledge, that's what I would do.

Also, unless source.txt is just a simple list of literals (which seems likely given the -F flag) you're probably going to have a bad time with 8MB of regexes. ripgrep is likely not sophisticated enough to handle it well. It might work with sufficient limits, but it really depends on your data.

In theory, given -F, this invocation should just use Aho-Corasick and not a regex engine at all. In which case, I believe the size limits shouldn't apply.

But again, without sharing the inputs here, it's hard to say.

[BurntSushi]

OK, so I created this ticket: #2247

ripgrep has an optimization for this case, but the -x/--line-regexp flag is inhibiting it. Fixing that bug is tricky.

With that said, if you're using -F and -x, then I imagine you could do even better by sorting both of your inputs and using comm.

[BurntSushi]

You can increase the size limits here, which will let ripgrep run, but it's going to be very slow. Usages like this bascially require that the Aho-Corasick optimization kick in. (Which is likely what happens with GNU grep. It suffers a similarly catastrophic slow-down if you remove the -F flag.)

[T145]

I've experimented w/ comm, but it requires input to be sorted and apparently parsort output isn't in a format that it likes. That may need to be revisited but other options such as ripgrep are what I'm hoping can give better performance. Are there other grep operations that can be used to accomplish this same task?

[BurntSushi]

GNU grep handles this particular case well.

Otherwise, I'm not sure there's anything else I can say. comm is what I would use here since it seems like it could do things most efficiently. Dunno why you're worrying about sorting when your data is so small. Just use regular sort.

[T145]

I ended up using parallel to handle speeding up the greping: handles it quite well! Best of luck on your bugfix.