Dictionary mismatch

[necros2k7]

Describe the bug
"zstd -d "1.warc.zst"
1.warc.zst : 0 B... 1.warc.zst : Decoding error (36) : Dictionary mismatch"
I guess some kind of dictionary to unpack should be used? Where to get it?

To Reproduce
Steps to reproduce the behavior:
try to decompress any "zstd -d "1.warc.zst"

Expected behavior
normal unpacking

Desktop (please complete the following information):

• OS: Win11
• Version 22H2
• Other relevant hardware specs X79 system

[Cyan4973]

I guess some kind of dictionary to unpack should be used? Where to get it?

We have no idea.

There is no managed repository of public dictionaries at the time of this writing,
consequently, all dictionaries are private. Therefore, it could be anything.

The dictionary ID can be consulted, using the command zstd -lv file.zst, though it will just provide an integer value.
This integer value is supposed to represent a (hopefully) unique file in the private referential of the compressor.
But that's about it, it doesn't tell much about the potential content of such a dictionary.
If one doesn't know where to retrieve the dictionary, chances are, the file can't be decompressed.

[JustAnotherArchivist]

Ugh, sorry about this spilling over to here.

@necros2k7 .warc.zst uses a custom format that includes the dictionary in the file itself, but regular zstd tooling doesn't know how to extract it. I wrote a little wrapper script to handle that.

For the record, the relevant specification is at http://iipc.github.io/warc-specifications/specifications/warc-zstd/, and the stalled attempt to upstream the embedded dictionary format is at #2349.

[necros2k7]

@JustAnotherArchivist , can you mod this script for Win?

[JustAnotherArchivist]

I don't see a particular reason why changes would be necessary, but if so, no, I'm afraid I don't have time for that. Maybe Windows Subsystem for Linux is an option. But that's all the advice I can give.