You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Column pruning minimizes I/O by reading only the columns used in a query and ignoring unused ones at the storage layer.
57
+
In Fluss, column pruning is implemented using [Apache Arrow](https://arrow.apache.org/) as the default log format to optimize streaming reads from Log Tables and change logs of PrimaryKey Tables.
58
+
Benchmark results show that column pruning can reach 10x read performance improvement, and reduce unnecessary network traffic (reduce 80% I/O if 80% columns are not used).
59
+
60
+
:::note
61
+
1. Column pruning is only available when the table uses the Arrow log format (`'table.log.format' = 'arrow'`), which is enabled by default.
62
+
2. Reading log data from remote storage currently does not support column pruning.
This confirms that only the `c_name` column is being read from storage.
99
+
100
+
### Partition Pruning
101
+
102
+
Partition pruning is an optimization technique for Fluss partitioned tables. It reduces the number of partitions scanned during a query by filtering based on partition keys.
103
+
This optimization is especially useful in streaming scenarios for [Multi-Field Partitioned Tables](table-design/data-distribution/partitioning.md#multi-field-partitioned-tables) that has many partitions.
104
+
The partition pruning also supports dynamically pruning new created partitions during streaming read.
105
+
106
+
:::note
107
+
1. Currently, **only equality conditions** (e.g., `c_nationkey = 'US'`) are supported for partition pruning. Operators like `<`, `>`, `OR`, and `IN` are not yet supported.
Fluss source will scan only the partitions where `c_nationkey = 'US'`.
133
+
For example, if the following partitions exist:
134
+
-`US,2025-06-13`
135
+
-`China,2025-06-13`
136
+
-`US,2025-06-14`
137
+
-`China,2025-06-14`
138
+
139
+
Only `US,2025-06-13` and `US,2025-06-14` will be read.
140
+
141
+
As new partitions like `US,2025-06-15`, `China,2025-06-15` are created, partition `US,2025-06-15` will be automatically included in the stream, while `China,2025-06-15` will be dynamically filtered out based on the partition pruning condition.
0 commit comments