Support getting block info for decompression

[skyguard1]

Is your feature request related to a problem? Please describe.
I am developing a ZstdDecoder on netty, network data packets are transmitted in blocks, and netty as a framework, in order to achieve universality, developers cannot be required to be compatible with our protocol. Therefore, is it possible to provide a method to determine the size of compressed data blocks during block decompression? I use the zstd-jni, and submitted a issue, but this repo cannot support this because zstd library does not expose any block info. Is this possible to provide this for decompression?

Describe the solution you'd like
Support getting block info for decompression

Describe alternatives you've considered
It is not a good way to judge the size of the compressed package after failing to decompress, the best solution is providing the block info including Last_Block and Block_Size for decompression

Additional context
I see that the block header has Last_Block and Block_Size for Compressed_Block in RFC-8478, Is this possible to add this?
Issue:
luben/zstd-jni#271
Thanks a lot

[Cyan4973]

I'm not sure to understand the question.
When stating : "determine the size of compressed data blocks during block decompression",
do you mean the size of the compressed block, therefore the compressed size,
or the size of the block after it's decompressed ?

[skyguard1]

Hi @Cyan4973
Thanks for the reply
My requirement is to provide the compressed size when decompressing to determine whether it is a complete compressed data block. I am adding a ZstdDecoder using zstd-jni, but the author of zstd-jni said that the zstd library does not support getting Last_Block and Block_Size for Compressed_Block ,I don't know much about the zstd library, but I saw that there are regulations in RFC-8478 that Last_Block and Block_Size for Compressed_Block, so I think it is possible to provide a method to expose block information in order to get the size of the compressed data block or isLastBlock
For more information, you can see luben/zstd-jni#271
Thanks

[Cyan4973]

So, it seems you want access to the information already present into the block's header.

The block header is indeed specified in the RFC8878, so its format is rigid. Any third-party can then use this specification in order to create its own decoder or its own block header reader.

If the question is about having an entry point in the reference library for a user to extract this information, note that this capability currently does not exist, and is not yet planned (you are the first to issue such request). The existing streaming API does provide a signal at end-of-frame, but not at end-of-block.

Importantly, it seems your assumption is that users are going to provide full blocks and only full blocks to the decoders, and naturally stop between block boundaries, and therefore ask block properties only at these specific boundaries. This might be true in your use case, but it's not true in general. In the large majority of cases, users send whatever data they have received, which can end up in the middle of a block. Therefore, an API requesting block header information must also be compatible this scenario, and provide a consistent behavior.

Finally, while it seems technically possible to create such an API, be aware that it will take time, and it will not reach "stable" status immediately. It's likely going to start its life in the "unstable/experimental" section, and iterate from there, using field experience.

[skyguard1]

Appreciate it. I think we can start with new feature and reach a steady state. In this scenario, we will determine the size of the compressed data block, and if the user does not provide a complete compressed data block, the ZstdDecoder will wait, but there will be exception handling when data block is not completed, so this is not a problem. So do you have plans to support it in future versions? Thanks a lot

[Cyan4973]

We could consider this item for next release.