Skip to content

Commit e51fcda

Browse files
lsyldliu风离polyzos
authored
[docs] Add docs for flink materialized table (#1798)
* [docs] Add docs for flink materialized table * small improvements --------- Co-authored-by: 风离 <[email protected]> Co-authored-by: ipolyzos <[email protected]>
1 parent 8c4004c commit e51fcda

File tree

2 files changed

+123
-17
lines changed

2 files changed

+123
-17
lines changed

‎website/docs/engine-flink/ddl.md‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -273,3 +273,106 @@ ALTER TABLE my_multi_fields_part_log_table DROP PARTITION (dt = '2025-03-05', na
273273
```
274274

275275
For more details, refer to the [Flink ALTER TABLE(DROP)](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/alter/#drop) documentation.
276+
277+
## Materialized Table
278+
### Overview
279+
Flink Materialized Table is a new table type introduced in Flink SQL that simplifies the development of both batch and streaming data pipelines.
280+
By defining the data freshness and query during creation, Flink automatically derives the table schema and generates a refresh pipeline to maintain the desired freshness level. This provides a unified and consistent development experience for both real-time and batch workloads. For more information, see the [Flink Materialized Table](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/materialized-table/overview) documentation.
281+
282+
Starting from Fluss version 0.8, Flink Materialized Table is now supported, which can significantly reduce the cost of building real-time data pipelines with Apache Flink and Fluss. Materialized tables in Fluss are implemented as regular Fluss tables with special metadata to identify them as materialized tables.
283+
284+
### Create Materialized Table
285+
286+
Materialized tables are created using the `CREATE MATERIALIZED TABLE` statement with a freshness interval and a query definition:
287+
288+
```sql title="Flink SQL"
289+
CREATE MATERIALIZED TABLE shop_summary
290+
FRESHNESS = INTERVAL '5' SECOND
291+
AS SELECT
292+
DATE_FORMAT(order_time, 'yyyy-MM-dd') AS order_date,
293+
shop_id,
294+
COUNT(*) AS order_count,
295+
SUM(amount) AS total_amount
296+
FROM orders
297+
GROUP BY DATE_FORMAT(order_time, 'yyyy-MM-dd'), shop_id;
298+
```
299+
300+
#### Supported Refresh Modes
301+
302+
Apache Fluss currently supports **CONTINUOUS** refresh mode for materialized table, which means the materialized table is continuously refreshed. The **FULL** refresh mode will be supported in future releases.
303+
304+
#### Schema Definition
305+
306+
The schema of a materialized table is automatically inferred from the query definition. You cannot manually specify column names and types - they are derived from the SELECT statement.
307+
308+
```sql title="Flink SQL"
309+
-- The schema will be automatically inferred as:
310+
-- order_date: STRING
311+
-- shop_id: BIGINT
312+
-- order_count: BIGINT
313+
-- total_amount: BIGINT
314+
CREATE MATERIALIZED TABLE daily_sales
315+
FRESHNESS = INTERVAL '1' MINUTE
316+
AS SELECT
317+
DATE_FORMAT(created_at, 'yyyy-MM-dd') AS order_date,
318+
shop_id,
319+
COUNT(*) AS order_count,
320+
SUM(amount) AS total_amount
321+
FROM sales_events
322+
GROUP BY DATE_FORMAT(created_at, 'yyyy-MM-dd'), shop_id;
323+
```
324+
325+
### Alter Materialized Table
326+
327+
You can suspend and resume materialized tables to control their refresh behavior:
328+
329+
#### Suspend Materialized Table
330+
331+
```sql title="Flink SQL"
332+
ALTER MATERIALIZED TABLE shop_summary SUSPEND;
333+
```
334+
335+
This stops the automatic refresh of the materialized table and saves the current state.
336+
337+
#### Resume Materialized Table
338+
339+
```sql title="Flink SQL"
340+
ALTER MATERIALIZED TABLE shop_summary RESUME;
341+
```
342+
343+
This resumes the automatic refresh of the materialized table from the last saved state.
344+
345+
### Drop Materialized Table
346+
347+
To delete a materialized table:
348+
349+
```sql title="Flink SQL"
350+
DROP MATERIALIZED TABLE shop_summary;
351+
```
352+
353+
This will drop the materialized table and stop the background refresh job.
354+
355+
### Materialized Table Options
356+
357+
Materialized tables support the same table options as regular Fluss tables, including partitioning and bucketing:
358+
359+
```sql title="Flink SQL"
360+
CREATE MATERIALIZED TABLE partitioned_summary
361+
FRESHNESS = INTERVAL '10' SECOND
362+
AS SELECT
363+
dt,
364+
shop_id,
365+
COUNT(*) AS order_count
366+
FROM orders
367+
GROUP BY dt, shop_id
368+
PARTITIONED BY (dt)
369+
WITH (
370+
'bucket.num' = '4'
371+
);
372+
```
373+
374+
### Limitations
375+
376+
- Only continuous refresh mode is supported
377+
- Schema is automatically derived from the query
378+
- Materialized tables are stored as regular Fluss tables with special metadata

‎website/docs/engine-flink/getting-started.md‎

Lines changed: 20 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ sidebar_position: 1
99
For a quick introduction to running Flink, refer to the [Quick Start](quickstart/flink.md) guide.
1010

1111

12-
## Support Flink Versions
12+
## Supported Flink Versions
1313
| Fluss Connector Versions | Supported Flink Versions |
1414
|--------------------------|--------------------------|
1515
| $FLUSS_VERSION_SHORT$ | 1.18, 1.19, 1.20 |
@@ -20,23 +20,26 @@ Fluss supports Apache Flink's Table API and Flink's DataStream API.
2020

2121
For Flink's Table API, Fluss supports the following features:
2222

23-
| Feature support | Flink | Notes |
23+
| Feature Support | Flink | Notes |
2424
|---------------------------------------------------|-------|----------------------------------------|
25-
| [SQL create catalog](ddl.md#create-catalog) | ✔️ | |
26-
| [SQL create database](ddl.md#create-database) | ✔️ | |
27-
| [SQL drop database](ddl.md#drop-database) | ✔️ | |
28-
| [SQL create table](ddl.md#create-table) | ✔️ | |
29-
| [SQL create table like](ddl.md#create-table-like) | ✔️ | |
30-
| [SQL drop table](ddl.md#drop-table) | ✔️ | |
31-
| [SQL show partitions](ddl.md#show-partitions) | ✔️ | |
32-
| [SQL add partition](ddl.md#add-partition) | ✔️ | |
33-
| [SQL drop partition](ddl.md#drop-partition) | ✔️ | |
34-
| [SQL select](reads.md) | ✔️ | Support both streaming and batch mode. |
35-
| [SQL limit](reads.md#limit-read) | ✔️ | Only for Log Table |
36-
| [SQL insert into](writes.md) | ✔️ | Support both streaming and batch mode. |
37-
| [SQL delete from](writes.md#delete-from) | ✔️ | Only in batch mode. |
38-
| [SQL update](writes.md#update) | ✔️ | Only in batch mode. |
39-
| [SQL lookup join](lookups.md) | ✔️ | |
25+
| [SQL Create Catalog](ddl.md#create-catalog) | ✔️ | |
26+
| [SQL Create Database](ddl.md#create-database) | ✔️ | |
27+
| [SQL Drop Database](ddl.md#drop-database) | ✔️ | |
28+
| [SQL Create Table](ddl.md#create-table) | ✔️ | |
29+
| [SQL Create Table Like](ddl.md#create-table-like) | ✔️ | |
30+
| [SQL Drop Table](ddl.md#drop-table) | ✔️ | |
31+
| [SQL Create Materialized Table](ddl.md#materialized-table) | ✔️ | Continuous refresh mode only |
32+
| [SQL Alter Materialized Table](ddl.md#alter-materialized-table) | ✔️ | Suspend/Resume support |
33+
| [SQL Drop Materialized Table](ddl.md#drop-materialized-table) | ✔️ | |
34+
| [SQL Show Partitions](ddl.md#show-partitions) | ✔️ | |
35+
| [SQL Add Partition](ddl.md#add-partition) | ✔️ | |
36+
| [SQL Drop Partition](ddl.md#drop-partition) | ✔️ | |
37+
| [SQL Select](reads.md) | ✔️ | Support both streaming and batch mode. |
38+
| [SQL Limit](reads.md#limit-read) | ✔️ | Only for Log Table |
39+
| [SQL Insert Into](writes.md) | ✔️ | Support both streaming and batch mode. |
40+
| [SQL Delete From](writes.md#delete-from) | ✔️ | Only in batch mode. |
41+
| [SQL Update](writes.md#update) | ✔️ | Only in batch mode. |
42+
| [SQL Lookup Join](lookups.md) | ✔️ | |
4043

4144
For Flink's DataStream API, you can see [DataStream API](docs/engine-flink/datastream.mdx) for more details.
4245

0 commit comments

Comments
 (0)