Environment variable for --memory

[catleeball]

Is your feature request related to a problem? Please describe.

Sometimes I use tools that utilize zstd, but they don't allow me to pass arbitrary flags to zstd (in this case rg -z). In these cases, the tool won't work in cases where the archive requires additional flags to decompress (such as needing --long=32 or --memory=2048MB)

Describe the solution you'd like

It would be great if there was an environment variable like ZSTD_MEMORY or similar to specify window size for compression/decompression.

This would allow me to use tools where I'm not able to pass options to zstd to decompress files that would need a flag like --long=32 or --memory=2048MB.

Describe alternatives you've considered

In this case, I wrote a shell script using xargs that ran zstdcat --long=31 on files and piped them to rg.

Additional context

In this case, I was using rg to gather statistics on a large compressed text corpora. Usually rg is nice for search a lot of compressed files in a directory, but these had been compressed with --long, so I got an error[1].

I also opened a bug in the ripgrep repo to support passing flags to the compression libraries it uses. That said, this would be a really nice general feature of zstd.

Thank you! :)

Footnotes

[1] Error message from rg, bubbling up an error from zstd:

-------------------------------------------------------------------------------                                                                                                               
./submissions/RS_2018-01.zst:                                                                                                                                                                 
-------------------------------------------------------------------------------                                                                                                               
sions/RS_2018-01.zst : Decoding error (36) : Frame requires too much memory for decoding                                                                                                      
sions/RS_2018-01.zst : Window size larger than maximum : 2147483648 > 134217728                                                                                                               
sions/RS_2018-01.zst : Use --long=31 or --memory=2048MB                                                                                                                                       
------------------------------------------------------------------------------- 

[Cyan4973]

We could easily solve this issue by removing any limitation (or selecting a larger limit) for the Window Size at decompression time.

While I'm sure it would better fit your use case, the problem is, in other environments, this could be considered a DoS attack vector: a crafted input could instruct the decompressor to allocate a vast amount of memory, with potentially some nasty consequences (we mitigate the worst of it by only reserving the memory address space, and the physical memory itself is normally not used until this amount of data is actually produced, but still).
More security conscious environments will not like such a possibility.

Our proposed solution is to offer a flag so that users can explicitly "opt-in" to employ more memory on their system when they want to.

But it seems this solution is not appropriate for your use case.

Now your proposition would be to change the default behavior, but as you can guess, this is always going to be in tension with other environments which would prefer other defaults.

The next solution that comes to mind is to provide some control through environment variables.
https://github.com/facebook/zstd/tree/v1.5.6/programs#passing-parameters-through-environment-variables
This is used precisely in these situations where the zstd CLI is indirectly invoked and there is no way to pass parameters via the intermediate caller.
There are already a few variables defined there, one idea could be to add another one, for example ZSTD_MEMORY, that would then change the default window size that the zstd decompressor accepts to allocate without requiring an explicit authorization. It could be extended or restricted, depending on your locally preferred default. And of course, the command line could still be used to step off those boundaries on demand.

[catleeball]

An environment variable for memory would be great too! I didn't know about the DoS attack vector, thanks for noting that.

I'll go ahead and edit the tile and comment to be an FR to be for an environment variable instead.

[gcflymoto]

Our proposed solution is to offer a flag so that users can explicitly "opt-in" to employ more memory on their system when they want to.

Is the opt-in during compilation time of zstd CLI utilities and libraries? That would be very helpful for our own fork and releases of zstd to opt-in to --long=31 support by default. However other tools and libraries in the zstd ecosystem that integrate with zstd may not opt-in and may make separate decisions. Excellent libraries like #pycompression/xopen#155 python have opted-in by default to support --long=31. Removing the friction, increasing flexibility and support for a variety of .zstd compressed inputs. However each system, application, libraries will have their own motives and reasons to choose not to opt-in. As much as I prefer not have any environmental control of zstd ZLI or libraries I don't see how to influence deeply nested systems which use zstd cli or libs without ZSTD_MEMORY or $HOME/.zstdrc (each come with their own set of encumbrances)

in other environments, this could be considered a DoS attack vector: a crafted input could instruct the decompressor to allocate a vast amount of memory, with potentially some nasty consequences

Any thoughts on other ways to mitigate this DoS attack vector? As internal automatic support for --long mode would still be the best all around solution.