modules/zstd: Add support for decoding compressed blocks #1857

lpawelcz · 2025-01-15T16:32:34Z

This PR extends the ZstdDecoder with support for decoding compressed blocks.

It supersedes PRs:

The decoder is capable of decoding RAW and RLE literals as well as sequences with predefined FSE tables.
A suite of DSLX tests comprising unit tests of all underlying procs and an integration test was prepared.
The integration test, similarly as in #1654, first generates a random valid ZSTD frame with compressed blocks and expected decoded output. Test data is then converted to a DSLX file (example) that is imported by the integration tests file.
At the beginning of the test, the default FSE decoding tables are filled with default distributions taken from RFC 8878 section 3.1.1.3.2.2. Default Distributions . Next, the encoded frame is loaded to the system memory and the decoder is configured through a set of CSRs to start the decoding process. The decoder starts the operation and writes the decoded frame back into the output buffer in the system memory. Once it finishes, it sends a pulse on the notify channel signaling the end of the decoding. The output of the decoder is compared against the decoding result from the reference library.

The PR introduces among others:

CompressedBlockDecoder - manages both SequenceDecoder and LiteralsDecoder to enable compress block decoding. Integrated with the top-level ZstdDecoder
SequenceDecoder - responsible for decoding sequence sections of the compressed blocks
FseDecoder - introduced as the core part of the SequenceDecoder
RefillingShiftBuffer - used for storing and outputting in forward and backward fashion an arbitrary amount of bits required by the FSE decoder
LiteralsDecoder - capable of decoding RAW, RLE and Huffman-coded literals
HuffmanDecoder - used in decoding huffman-coded literals. Decoded Huffman trees are then used to decode one or four Huffman-coded streams.
CommandConstructor - this proc is responsible for sending packets with decoded sequences and literals to the SequenceExecutor proc
RamMux and RamDemux - procs used for handling requests/responses to multiple memory models. The procs interface with 3 separate memory buffers for FSE decoding tables.

google-cla · 2025-01-15T16:32:43Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

proppy · 2025-01-27T01:25:00Z

Can we rebase and consolidate with #1654 ?

lpawelcz · 2025-02-05T09:15:24Z

Can we rebase and consolidate with #1654 ?

@proppy done
This PR also supersedes #1616 so I also closed it in favor of this one

proppy · 2025-03-06T08:16:09Z

xls/dslx/stdlib/BUILD

@@ -59,6 +59,11 @@ xls_dslx_prove_quickcheck_test(
    srcs = ["std.x"],
 )

+xls_dslx_library(


can we get a separate PR for that?

proppy · 2025-03-06T08:16:44Z

xls/modules/zstd/zstd_frame_dslx.py

@@ -0,0 +1,170 @@
+# Copyright 2024 The XLS Authors


can. you add a docstring about what the module is doing and where it is being used?

proppy · 2025-03-06T08:17:37Z

xls/modules/zstd/external/BUILD

@@ -0,0 +1,33 @@
+# Copyright 2024 The XLS Authors


can. you add a docstring about what those external are and why they are needed?

proppy · 2025-03-06T08:18:08Z

xls/modules/rle/rle_common.x

@@ -24,6 +26,15 @@ pub struct PlainData<SYMB_WIDTH: u32> {
    last: bool,               // flush RLE
 }

+// Structure contains multiple uncompressed symbols.


is that used by zstd?

proppy · 2025-03-06T08:18:56Z

xls/modules/zstd/zstd_dec_wrapper.v

@@ -0,0 +1,753 @@
+// Copyright 2024 The XLS Authors


can. you add a docstring about what the module is and why it is needed? (if it's for driving the cocostb test maybe move it there or in a separate rtl folder?)

proppy · 2025-03-06T08:19:37Z

xls/modules/zstd/memory/mem_writer_wrapper.v

@@ -0,0 +1,193 @@
+// Copyright 2024 The XLS Authors


can you move the verilog file in a separate rtl directory w/ a README (or docstring describing the modules)?

proppy · 2025-03-06T08:20:11Z

can you also rebase?

- XLSStruct for easier handling and serializing/deserializing XLS structs - XLSChannel that serves as a dummy receiving channel - XLSMonitor that monitors transactions on an XLS channel - XLSDriver that can send data on an XLS channel - LatencyScoreboard that can measure latency between corresponding transactions on input and output buses - File-backed AXI memory python model Internal-tag: [#64075] Signed-off-by: Krzysztof Obłonczek <[email protected]>

Signed-off-by: Pawel Czarnecki <[email protected]>

Co-authred-by: Pawel Czarnecki <[email protected]> Co-authred-by: Robert Winkler <[email protected]> Signed-off-by: Michal Czyz <[email protected]> Signed-off-by: Pawel Czarnecki <[email protected]> Signed-off-by: Robert Winkler <[email protected]>

Signed-off-by: Pawel Czarnecki <[email protected]>

Signed-off-by: Maciej Torhan <[email protected]>

Signed-off-by: Pawel Czarnecki <[email protected]>

Signed-off-by: Robert Winkler <[email protected]>

Signed-off-by: Pawel Czarnecki <[email protected]>

Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>

Signed-off-by: Robert Winkler <[email protected]>

Signed-off-by: Maciej Torhan <[email protected]>

Signed-off-by: Robert Winkler <[email protected]>

Internal-tag: [#71197]

Signed-off-by: Robert Winkler <[email protected]>

Internal-tag: [#71791] Signed-off-by: Pawel Czarnecki <[email protected]>

Co-authored-by: Krzysztof Obłonczek <[email protected]> Signed-off-by: Robert Winkler <[email protected]>

Signed-off-by: Robert Winkler <[email protected]>

Internal-tag: [#71874] Signed-off-by: Pawel Czarnecki <[email protected]>

Signed-off-by: Pawel Czarnecki <[email protected]>

Internal-tag: [#71766] Signed-off-by: Pawel Czarnecki <[email protected]>

Co-authored-by: Krzysztof Obłonczek <[email protected]> Signed-off-by: Maciej Torhan <[email protected]>

Signed-off-by: Robert Winkler <[email protected]>

tmichalak force-pushed the zstd_compressed_block_dec branch from 47ab08c to 50f6140 Compare January 15, 2025 20:51

lpawelcz force-pushed the zstd_compressed_block_dec branch from 50f6140 to 36bc838 Compare January 16, 2025 07:19

This was referenced Jan 20, 2025

modules/zstd: Add sequence decoding #1362

Closed

modules/zstd: Add literals decoding #1430

Closed

proppy mentioned this pull request Jan 27, 2025

modules/zstd: Rework ZSTD Decoder #1654

Closed

7 tasks

rw1nkler force-pushed the zstd_compressed_block_dec branch from 36bc838 to 16cd455 Compare January 29, 2025 21:28

rw1nkler mentioned this pull request Jan 30, 2025

Huge RAM consumption in DSLX interpreter #1897

Open

lpawelcz mentioned this pull request Feb 5, 2025

Add cocotb tests for AXI peripherals #1616

Closed

proppy reviewed Mar 6, 2025

View reviewed changes

koblonczek and others added 12 commits March 26, 2025 22:59

modules/zstd/memory/MemReader: Add cocotb tests

1f66299

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/memory/MemWriter: Add cocotb test

0ac6ce7

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/memory/README: Describe verilog simulation

3d8676a

Signed-off-by: Pawel Czarnecki <[email protected]>

CI/modules-zstd: Add calls to verilog simulation targets

a8e178d

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/BUILD: increase pipeline_stages for DecoderMux proc

ea2c895

Signed-off-by: Pawel Czarnecki <[email protected]>

xls/modules/zstd: expose fifo verilog module

7986368

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/rle/common: Add RLE output struct

07f4e87

Signed-off-by: Maciej Torhan <[email protected]>

modules/zstd: Remove MagicNumberDecoder

3c6864b

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd: Remove BlockDecoder proc

ecdee67

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd: Remove DecDemux proc

f5810ba

Signed-off-by: Pawel Czarnecki <[email protected]>

rw1nkler and others added 29 commits March 26, 2025 23:07

modules/zstd/BUILD: tag targets as manual

af64bff

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd: Extend common library

365b851

Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd: Enable CompressedBlockDecoder

a8b87d6

Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>

modules/zstd: Enable CompressedBlockDecoder in ZstdDecoder

da278e3

Co-authored-by: Robert Winkler <[email protected]> Signed-off-by: Robert Winkler <[email protected]> Signed-off-by: Krzysztof Obłonczek <[email protected]>

modules/zstd: Add example frame for DSLX tests

c62c5c8

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd: Adjust SequenceExecutor

5ddcdad

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd: Adjust AxiRam

dea4598

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd: Add RamMux

02286c5

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd: Fix HuffmanRawWeightsDecoder

f95385d

Signed-off-by: Maciej Torhan <[email protected]>

Fix decoding multiple frames

0e06068

Signed-off-by: Robert Winkler <[email protected]>

Add support for Compressed FSE tables

7771e3a

Internal-tag: [#71197]

Add test frame for compressed fse tables

1b2aa15

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd/zstd_dec: don't use channel arrays for FSE tables access

4f41fab

Internal-tag: [#71791] Signed-off-by: Pawel Czarnecki <[email protected]>

Fix sequence decoding with predefined offset table, add regression test

500b5b3

Add support for RLE and Repeated fse tables

b268895

Co-authored-by: Krzysztof Obłonczek <[email protected]> Signed-off-by: Robert Winkler <[email protected]>

Add test frame for repeated and rle tables

3b8b65b

Signed-off-by: Robert Winkler <[email protected]>

modules/zstd/BUILD: Add RAM ports rewrite configuration for ZstdDecoder

e660736

Internal-tag: [#71874] Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd/zstd_dec_wrapper: Remove DSLX native RAM ports

5abf36f

Internal-tag: [#71874] Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd: Add RamMerge proc

daaa66c

Internal-tag: [#71874] Signed-off-by: Pawel Czarnecki <[email protected]>

[TEMP] modules/zstd/zstd_dec: Inline HuffmanWeights RamModel

50c6aa1

Signed-off-by: Pawel Czarnecki <[email protected]>

[TEMP] modules/zstd/BUILD: Disable ZstdDecoder P&R

d03badb

Internal-tag: [#71766] Signed-off-by: Pawel Czarnecki <[email protected]>

modules/zstd: Implement HuffmanFseWeightsDecoder

0965dfa

Co-authored-by: Krzysztof Obłonczek <[email protected]> Signed-off-by: Maciej Torhan <[email protected]>

Count the length of the section with probability frequencies

004a141

Signed-off-by: Robert Winkler <[email protected]>

Adjust the design to the changed HuffmanWeightsDecoder

ee9d0ee

Signed-off-by: Robert Winkler <[email protected]>

Remove duplicate fifo.v file

2c120e0

Signed-off-by: Robert Winkler <[email protected]>

Move test files to a separate directory

22c9474

Signed-off-by: Robert Winkler <[email protected]>

Move shift_buffer to zstd directory

46f559c

Signed-off-by: Robert Winkler <[email protected]>

Adjust BUILD files to the recent changes in the toolchain

9b44145

Signed-off-by: Robert Winkler <[email protected]>

Adjust design to the recent changes in the toolchain

cae18b3

Signed-off-by: Robert Winkler <[email protected]>

rw1nkler force-pushed the zstd_compressed_block_dec branch from 16cd455 to cae18b3 Compare March 27, 2025 06:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modules/zstd: Add support for decoding compressed blocks #1857

modules/zstd: Add support for decoding compressed blocks #1857

lpawelcz commented Jan 15, 2025 •

edited

Loading

google-cla bot commented Jan 15, 2025

proppy commented Jan 27, 2025

lpawelcz commented Feb 5, 2025

proppy Mar 6, 2025

proppy Mar 6, 2025 •

edited

Loading

proppy Mar 6, 2025

proppy Mar 6, 2025

proppy Mar 6, 2025

proppy Mar 6, 2025

proppy commented Mar 6, 2025

modules/zstd: Add support for decoding compressed blocks #1857

Are you sure you want to change the base?

modules/zstd: Add support for decoding compressed blocks #1857

Conversation

lpawelcz commented Jan 15, 2025 • edited Loading

google-cla bot commented Jan 15, 2025

proppy commented Jan 27, 2025

lpawelcz commented Feb 5, 2025

proppy Mar 6, 2025

Choose a reason for hiding this comment

proppy Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

proppy Mar 6, 2025

Choose a reason for hiding this comment

proppy Mar 6, 2025

Choose a reason for hiding this comment

proppy Mar 6, 2025

Choose a reason for hiding this comment

proppy Mar 6, 2025

Choose a reason for hiding this comment

proppy commented Mar 6, 2025

lpawelcz commented Jan 15, 2025 •

edited

Loading

proppy Mar 6, 2025 •

edited

Loading