Skip to content

Commit 613d354

Browse files
authored
[lake/lance] add Flink memory usage note (#1909)
1 parent fcd2378 commit 613d354

File tree

1 file changed

+2
-0
lines changed
  • website/docs/streaming-lakehouse/integrate-data-lakes

1 file changed

+2
-0
lines changed

website/docs/streaming-lakehouse/integrate-data-lakes/lance.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,8 @@ Additionally, when following the [Start Datalake Tiering Service](maintenance/ti
7272

7373
> **NOTE**: Fluss v0.8 only supports tiering log tables to Lance.
7474
75+
> **NOTE**: The Lance connector leverages Arrow Java library, which operates on off-heap memory. To prevent `java.lang.OutOfMemoryError: Direct buffer memory` error in Flink Task Manager, please increase the value of `taskmanager.memory.task.off-heap.size` in `<FLINK_HOME>/conf/config.yaml` to at least `'512m'` (e.g., `taskmanager.memory.task.off-heap.size: 512m`). You may need to adjust this value higher (such as `'1g'`) depending on your workload and data size.
76+
7577
Then, the datalake tiering service continuously tiers data from Fluss to Lance. The parameter `table.datalake.freshness` controls the frequency that Fluss writes data to Lance tables. By default, the data freshness is 3 minutes.
7678

7779
You can also specify Lance table properties when creating a datalake-enabled Fluss table by using the `lance.` prefix within the Fluss table properties clause.

0 commit comments

Comments
 (0)