zstd dictionary generation and usage

[adminx01]

Hey there, to the never ending quest of getting better compression ratio, i've been looking into using dictionaries with zstd.

Problem is it takes ages to generate one even from a few mib, it seems to be designed for super-computers to generate? perhaps Facebook's? So I have not been able to generate one yet.

I am not even able to find generated dictionaries much, maybe someone could help with that.

I saw that dwarfs supports it but the command for it is not quite easy to understand. I think i figured it out some months ago but forgot since then haha

[mhx]

FWIW, DwarFS doesn't support supplying custom dictionaries to use with zstd (unless I've forgotten that I've implemented this).

[mhx]

Also, I'd doubt that you could save much by using an "optimized" dictionary. The main purpose of using custom dictionaries is reusing a shared dictionary for several small pieces of data, with a typical use case being client-server messaging. Client and server exchange a dictionary once and then use this shared dictionary for further communication, which can significantly reduce message size.

[adminx01]

oh, i didn't know.