Consider supporting tar file

[upsuper]

Opening this issue as suggested by @BurntSushi in #1111.

I think it would be helpful if ripgrep can support scanning archive files. Scanning tar files can briefly done via -a option, but there are downsides:

• you need to extract all files from the archive and scan again to identify the files you want
• the line number is unhelpfully large
• it may be wasting time on scanning files that we don't actually need

It is unclear how scanning archive files should work. It should definitely be opt-in just like --search-zip. We can probably add --search-archive or --search-tar for this purpose. I guess ideally archive files should probably be handled like directory rather than single file in that mode.

That should lead to some thing like

path/to/archive.tar/file1
something matched

path/to/archive.tar/file2
something else matched

@BurntSushi what do you think?

[BurntSushi]

Thanks for filing the ticket! I think this needs to be fleshed out quite a bit more though.

I guess ideally archive files should probably be handled like directory rather than single file in that mode.

ripgrep has fairly sophisticated filtering support. How does that interact with treating tar files as if they were a directory? e.g., If there is an applicable *.txt exclusion filter in play and a tar archive contains foo.txt, should that file also be ignored? If so, which filters are applied to it and how does that work? Personally, I suspect this will be a significant complication.

[upsuper]

For the case you mentioned, it should probably ignore the foo.txt, but still scan other files in the archives, just like what we do for directory.

For how does that work... so it seems currently this is done via ignore crate, but integrating this support into that crate is probably not an option, because we would need decompression support there as well. Maybe we can change how ignore work, that we walk through directories in ripgrep, and feed file path into ignore for whether a directory should be further traversed, and whether a file should be scanned. This effectively would split ignore into two crates, one for walking through file system, another for checking file inclusion. It would be a non-trivial refactor. Alternatively we can add some new API to ignore for querying this kind of information, probably.

[BurntSushi]

Right. This is probably blocked on refactoring and cleaning up the ignore crate in general. I don't expect that to happen any time soon, and I haven't had the time to write down enough of my thoughts on that for others to see.

I'll leave this open for now, but I don't think this is going to happen any time soon. Moreover, I am still not quite convinced that this feature belongs in ripgrep. It's definitely something that folks have requested before though.

[mateon1]

I suggest adding a generic --search-archive flag, so this can support different types of archives (like zip files) in the future.

This would be really helpful for me, as I often need to grep through a massive corpus of text, hundreds of gigabytes in size. If I use ripgrep on the unpacked corpus, I get very useful information like filenames of the files that matched, and real line numbers. Unfortunately, because of spinning disk random access times this search usually takes over 6 hours to process the whole corpus.
If I instead search the compressed version of the corpus by piping the output of unzip -p '*'.zip, the search only takes about an hour, but I lose information like which file a specific matching line came from, which sometimes forces me to repeat the search (or do a more specific search) on the uncompressed corpus later.

[genivia-inc]

My 2c.

I was looking for the same thing as I have several archived projects (.tar.gz) from years ago that I want to grep through fast without extracting files first. Eventually I decided to implement my own grep tool for this and other reasons. Recently I added tar file searches that I had on my TODO list, see ugrep for the latest. Be warned that ugrep is pretty new (since mid 2019). It is stable but there may still be some rough edges, though I've tested it regularly on large test cases and tar files without issues. Supports (gzip compressed) tar v7, ustar, gnu tar, and pax extended header blocks for long file names. So right now it covers the common use cases.

Anyway, if this is not helpful to ripgrep then I hope it may be helpful to you.

You can shoot me down for not helping this community to implement tar file search in a ripgrep fork, but I'm just old fashioned and mostly code C/C++ and I'm too lazy too learn Rust.

[eadmaster]

I've just found this wrapper that adds some compressed archives support.

Older alternatives i know are zipgrep and zzfind.

Btw, it is not clear to me how the current --search-zip flag is supposed to work, can you provide some examples?

[BurntSushi]

@eadmaster It's pretty simple. You give ripgrep compressed files and it searches them:

$ echo 'foo bar quux' > /tmp/haystack
$ rg bar /tmp/haystack
1:foo bar quux
$ gzip /tmp/haystack
$ rg bar /tmp/haystack.gz
$ rg -z bar /tmp/haystack.gz
1:foo bar quux

It sounds like you're trying to give ripgrep compressed archive files. ripgrep doesn't support iterating over archives, even when they're uncompressed. It's orthogonal to --search-zip.

[eadmaster]

I see, then the name search-zip is a bit ambiguous for me, why not changing to search-gz or search-gzip?

[BurntSushi]

Because the flag name is used in other tools, change causes churn and gzip is not the only supported compression format.