Problem description
I'm using Valkey GLIDE as my client to talk to a Valkey cluster. They have an (experimental) feature to enable compression https://glide.valkey.io/concepts/client-features/compression/, with either ZSTD or LZ4.
I thought that this would work great with the Enable Automatic Data Decompression -> ZSTD option. But it doesn't.
The reason is that GLIDE does this;
- A short magic header is prepended to identify the data as compressed and record which backend was used.
Which is defined here https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs#L1103
So a 5 byte magic header is prepended to the actual compressed data. Breaking the automatic compression detection in Redis Insight.
Once those 5 bytes are removed, decompression works as expected. I have tested this on the dev build against my ZSTD compressed data and it works. I have not tested against LZ4 which seems to also have 4 bytes representing the original size https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs?#L774-L777 before the actual data.
I'm thinking that the ZSTD | LZ4 could also check for these initial magic bytes and unpack before handing the data to the decompression fn. Or that a new option is introduced in the dropdown. Maybe just Valkey GLIDE? Since they encode in their envelope which backend was used. Which would then shift the data until we arrive att the compressed bytes and then hand them over to the existing decompression path.
Is this something you'd be interested in supporting? If so, any preference on how? I could take a look at implementing a solution if you want.
How often do you encounter this problem Every time I view data compressed by GLIDE.
Problem description
I'm using Valkey GLIDE as my client to talk to a Valkey cluster. They have an (experimental) feature to enable compression https://glide.valkey.io/concepts/client-features/compression/, with either
ZSTDorLZ4.I thought that this would work great with the
Enable Automatic Data Decompression -> ZSTDoption. But it doesn't.The reason is that GLIDE does this;
Which is defined here https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs#L1103
So a 5 byte magic header is prepended to the actual compressed data. Breaking the automatic compression detection in Redis Insight.
Once those 5 bytes are removed, decompression works as expected. I have tested this on the dev build against my
ZSTDcompressed data and it works. I have not tested againstLZ4which seems to also have 4 bytes representing the original size https://github.com/valkey-io/valkey-glide/blob/d3feeaaf08e0cb4b8eff17ea7df1ddbc6e6d0c73/glide-core/src/compression.rs?#L774-L777 before the actual data.I'm thinking that the
ZSTD | LZ4could also check for these initial magic bytes and unpack before handing the data to the decompression fn. Or that a new option is introduced in the dropdown. Maybe justValkey GLIDE? Since they encode in their envelope which backend was used. Which would then shift the data until we arrive att the compressed bytes and then hand them over to the existing decompression path.Is this something you'd be interested in supporting? If so, any preference on how? I could take a look at implementing a solution if you want.
How often do you encounter this problem Every time I view data compressed by GLIDE.