Support STDIN and STDOUT in CLI

[ibnesayeed]

The current version of the CLI makes it difficult to read/write data from/to STDIN/STDOUT. This means, one has to create named pipes (FIFO files) or temporary compressed or decompressed files on disk for further processing. The current CLI also makes it difficult to concatenate/append compressed or decompressed data from multiple sources (e.g., looping over multiple files) into a single file (with the --force flag it would overwrite existing data rather than appending).

Most of the standard compression tools (like gzip) allow reading from STDIN and writing to STDOUT, which plays well with simple pipes and redirects.

I have tried using /dev/stdin and /dev/stdout as input and output files, but in some circumstances this method does not work as expected (e.g., when trying to append the output to a single file).

Another quality of life improvement would be to introduce a --quiet flag to suppress any warnings or info printed on STDERR.

[felixhandte]

What operating system are you working on?

Can you be more specific about how /dev/stdin and /dev/stdout don't work? I guess the construction -o /dev/stdout >> existing_file_you_want_to_append_to doesn't append. Is that what you're talking about?

[ibnesayeed]

What operating system are you working on?

I am using Ubuntu 22.04 for testing.

Can you be more specific about how /dev/stdin and /dev/stdout don't work? I guess the construction -o /dev/stdout >> existing_file_you_want_to_append_to doesn't append. Is that what you're talking about?

Yes, this is one of the pain points. Below is a minimal example to reproduce it.

First, create two simple test input files:

$ echo "Line 1" > test-in-1.txt
$ echo "Line 2" > test-in-2.txt

Compress the two files one by one, both using /dev/stdout as the output file. This work as expected and prints binary data on STDOUT without complaining.

$ zli compress -p serial -o /dev/stdout test-in-1.txt
Compressed 7 -> 27 (0.26x) in 0.157 ms, 0.04 MiB/s
[BINARY_DATA]
         
$ zli compress -p serial -o /dev/stdout test-in-2.txt
Compressed 7 -> 27 (0.26x) in 0.190 ms, 0.04 MiB/s
[BINARY_DATA]

Now, redirect the output to a file, and it would throw and error about an already existing output file (though, the file does not exist when the command is run, but maybe it is created by the shell after interpreting the redirection), and exit with code 1.

$ zli compress -p serial -o /dev/stdout test-in-1.txt > test-out.txt.zl
Invalid argument(s):
        Output file already exists. Use --force to overwrite.

Now, add the --force flag to overwrite (in this case the output file is STDOUT, redirected to a desired combined compressed file) and try to append the second file to to it.

$ zli compress -p serial -o /dev/stdout -f test-in-1.txt > test-out.txt.zl
Compressed 7 -> 27 (0.26x) in 0.103 ms, 0.07 MiB/s

$ zli compress -p serial -o /dev/stdout -f test-in-2.txt >> test-out.txt.zl
Compressed 7 -> 27 (0.26x) in 0.098 ms, 0.07 MiB/s

Finally, decompress to see what was compressed.

$ zli decompress test-out.txt.zl
Decompressed: 385.71% (  27.00 B ->    7.00 B) in 0.429 ms, 0.02 MiB/s

$ cat test-out.txt
Line 2

The first input data, when compressed, was overwritten by the last one.

[Hanu-man12]

Hi @felixhandte, are you still working on this?
I'd be happy to help or take this over if needed.