Skip to content

Commit ae08d34

Browse files
swuferhongwuchong
authored andcommitted
[docs] Add document for Arrow compression
1 parent 3b04c85 commit ae08d34

File tree

1 file changed

+43
-0
lines changed

1 file changed

+43
-0
lines changed

website/docs/table-design/table-types/log-table.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,46 @@ Log Tables in Fluss allow real-time data consumption, preserving the order of da
4242
## Log Tiering
4343
Log Table supports tiering data to different storage tiers. See more details about [Remote Log](/docs/maintenance/tiered-storage/remote-storage/).
4444

45+
## Log Compression
46+
Log Table supports the end-to-end block compression feature for arrow log format. If enabled, data will be compressed
47+
by the writer in client, written in compressed format and decompressed by the log scanner in client. Log compression
48+
can significantly reduce storage costs on the server side.
49+
50+
Currently, Log Table is defaulted to using `ZSTD` compression codec, with the compression level set to `3`. If you want to
51+
modify the compression codec or compression level of `ZSTD` compression codec, you can create table as:
52+
53+
```sql title="Flink SQL"
54+
-- Set the compression codec to LZ4_FRAME
55+
CREATE TABLE log_table (
56+
order_id BIGINT,
57+
item_id BIGINT,
58+
amount INT,
59+
address STRING
60+
)
61+
WITH (
62+
'table.log.arrow.compression.type' = 'LZ4_FRAME'
63+
);
64+
65+
-- Set the 'ZSTD' compression level to 2
66+
CREATE TABLE log_table (
67+
order_id BIGINT,
68+
item_id BIGINT,
69+
amount INT,
70+
address STRING
71+
)
72+
WITH (
73+
'table.log.arrow.compression.zstd.level' = '2'
74+
);
75+
```
76+
In the above example, we set the compression codec to `LZ4_FRAME` and the compression level to `2`.
77+
78+
:::note
79+
1. Currently, the compression codec and compression level are only supported for arrow format. If you set `'table.log.format'='indexed'`, the compression codec and compression level will be ignored.
80+
2. The valid range of `table.log.arrow.compression.zstd.level` is 1 to 22.
81+
:::
82+
83+
### Supported Compression Codecs
84+
Currently, Fluss supports the following compression codecs for arrow format:
85+
- `NONE`: No compression.
86+
- `LZ4_FRAME`: LZ4 frame compression.
87+
- `ZSTD`: ZSTD compression.

0 commit comments

Comments
 (0)