Skip to content

Commit bbc3cda

Browse files
committed
add lance doc
1 parent 61aa7be commit bbc3cda

File tree

1 file changed

+9
-24
lines changed
  • website/docs/streaming-lakehouse/integrate-data-lakes

1 file changed

+9
-24
lines changed

website/docs/streaming-lakehouse/integrate-data-lakes/lance.md

Lines changed: 9 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,13 @@ sidebar_position: 1
55

66
# Lance
77

8-
[Apache Paimon](https://paimon.apache.org/) innovatively combines a lake format with an LSM (Log-Structured Merge-tree) structure, bringing efficient updates into the lake architecture.
8+
[Lance](https://lancedb.github.io/lance/) is a modern table format optimized for machine learning and AI applications.
99
To integrate Fluss with Lance, you must enable lakehouse storage and configure Lance as the lakehouse storage. For more details, see [Enable Lakehouse Storage](maintenance/tiered-storage/lakehouse-storage.md#enable-lakehouse-storage).
1010

1111
## Introduction
1212

1313
When a table is created or altered with the option `'table.datalake.enabled' = 'true'`, Fluss will automatically create a corresponding Lance table with the same table path.
14-
The schema of the Paimon table matches that of the Fluss table.
14+
The schema of the Lance table matches that of the Fluss table.
1515

1616
```sql title="Flink SQL"
1717
USE CATALOG fluss_catalog;
@@ -31,10 +31,9 @@ CREATE TABLE fluss_order_with_lake (
3131
);
3232
```
3333

34-
Then, the datalake tiering service continuously tiers data from Fluss to Lance. The parameter `table.datalake.freshness` controls the frequency that Fluss writes data to Paimon tables. By default, the data freshness is 3 minutes.
35-
For primary key tables, changelogs are also generated in the Paimon format, enabling stream-based consumption via Paimon APIs.
34+
Then, the datalake tiering service continuously tiers data from Fluss to Lance. The parameter `table.datalake.freshness` controls the frequency that Fluss writes data to Lance tables. By default, the data freshness is 3 minutes.
3635

37-
Since Fluss version 0.7, you can also specify Paimon table properties when creating a datalake-enabled Fluss table by using the `paimon.` prefix within the Fluss table properties clause.
36+
Since Fluss version 0.7, you can also specify Lance table properties when creating a datalake-enabled Fluss table by using the `lance.` prefix within the Fluss table properties clause.
3837

3938
```sql title="Flink SQL"
4039
CREATE TABLE fluss_order_with_lake (
@@ -54,13 +53,11 @@ CREATE TABLE fluss_order_with_lake (
5453
);
5554
```
5655

57-
For example, you can specify the Paimon property `file.format` to change the file format of the Paimon table, or set `deletion-vectors.enabled` to enable or disable deletion vectors for the Paimon table.
56+
For example, you can specify the Lance property `file.format` to change the file format of the Paimon table, or set `deletion-vectors.enabled` to enable or disable deletion vectors for the Paimon table.
5857

5958
### Reading with other Engines
6059

61-
Since the data tiered to Paimon from Fluss is stored as a standard Paimon table, you can use any engine that supports Paimon to read it. Below is an example using [StarRocks](https://paimon.apache.org/docs/master/engines/starrocks/):
62-
63-
First, create a Paimon catalog in StarRocks:
60+
Since the data tiered to Lance from Fluss is stored as a standard Lance table, you can use any engine that supports Lance to read it. Below is an example using [pylance](https://pypi.org/project/pylance/):
6461

6562
```sql title="StarRocks SQL"
6663
CREATE EXTERNAL CATALOG paimon_catalog
@@ -73,24 +70,12 @@ PROPERTIES (
7370

7471
> **NOTE**: The configuration values for `paimon.catalog.type` and `paimon.catalog.warehouse` must match those used when configuring Paimon as the lakehouse storage for Fluss in `server.yaml`.
7572
76-
Then, you can query the `orders` table using StarRocks:
77-
78-
```sql title="StarRocks SQL"
79-
-- The table is in the database `fluss`
80-
SELECT COUNT(*) FROM paimon_catalog.fluss.orders;
81-
```
82-
83-
```sql title="StarRocks SQL"
84-
-- Query the system tables to view snapshots of the table
85-
SELECT * FROM paimon_catalog.fluss.enriched_orders$snapshots;
86-
```
87-
8873
## Data Type Mapping
8974

90-
When integrating with Paimon, Fluss automatically converts between Fluss data types and Paimon data types.
91-
The following table shows the mapping between [Fluss data types](table-design/data-types.md) and Paimon data types:
75+
When integrating with Lance, Fluss automatically converts between Fluss data types and Lance data types.
76+
The following table shows the mapping between [Fluss data types](table-design/data-types.md) and Lance data types:
9277

93-
| Fluss Data Type | Paimon Data Type |
78+
| Fluss Data Type | Lance Data Type |
9479
|-------------------------------|-------------------------------|
9580
| BOOLEAN | BOOLEAN |
9681
| TINYINT | TINYINT |

0 commit comments

Comments
 (0)