Skip to content

Commit bb6fcbd

Browse files
committed
updates
1 parent f26743c commit bb6fcbd

File tree

1 file changed

+33
-21
lines changed
  • website/docs/table-design/table-types/pk-table/merge-engines

1 file changed

+33
-21
lines changed

website/docs/table-design/table-types/pk-table/merge-engines/versioned.md

Lines changed: 33 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,29 +5,41 @@ sidebar_position: 3
55

66
# Versioned Merge Engine
77

8-
By setting `'table.merge-engine' = 'versioned'`, users can update data based on the configured version column. Updates will be carried out when the latest value of the specified field is greater than or equal to the stored value. If it is less than or null, no update will be made.
9-
This feature is particularly valuable for replacing [Deduplication](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/deduplication/) transformations in streaming computations, reducing complexity and improving overall efficiency.
8+
The **Versioned Merge Engine** enables data updates based on version numbers or event timestamps. It ensures that only the row with the highest version number (or event timestamp) for a given primary key is retained. This mechanism is particularly useful for deduplicating or merging out-of-order data while guaranteeing eventual consistency with the upstream source.
109

11-
:::note
12-
When using `versioned` merge engine, there are the following limits:
13-
- `UPDATE` and `DELETE` statements are not supported.
14-
- Partial update is not supported.
15-
- `UPDATE_BEFORE` and `DELETE` changelog events are ignored automatically.
16-
:::
10+
By setting `'table.merge-engine' = 'versioned'`, users can update data based on a configured version column. Updates are performed when the latest value of the specified field is greater than or equal to the stored value. If the incoming value is less than the stored value or is null, no update will occur.
11+
12+
This feature is especially valuable as a replacement for [Deduplication](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/deduplication/) transformations in streaming computations. It simplifies workflows, reduces complexity, and improves overall efficiency.
1713

18-
## Versioned Merge Column
1914

2015
:::note
21-
The versioned merge column supports the following data types.
22-
- INT
23-
- BIGINT
24-
- TIMESTAMP
25-
- TIMESTAMP(p)
26-
- TIMESTAMP_LTZ
27-
- TIMESTAMP_LTZ(p)
16+
When using the `versioned` merge engine, keep the following limitations in mind:
17+
- **`UPDATE` and `DELETE` statements are not supported.**
18+
- **Partial updates are not supported.**
19+
- **`UPDATE_BEFORE` and `DELETE` changelog events are ignored automatically.**
2820
:::
2921

30-
example:
22+
### Version Column
23+
24+
The version column is a column in the table that stores the version number (or event timestamp) of the data record.
25+
When enabling the versioned merge engine, the version column must be explicitly specified using the property:
26+
27+
```sql
28+
'table.merge-engine' = 'versioned',
29+
'table.merge-engine.versioned.ver-column' = '<column_name>'
30+
```
31+
32+
The version column can be of the following data types:
33+
- `INT`
34+
- `BIGINT`
35+
- `TIMESTAMP`
36+
- `TIMESTAMP(p)` (with precision)
37+
- `TIMESTAMP_LTZ` (timestamp with local time zone)
38+
- `TIMESTAMP_LTZ(p)` (timestamp with local time zone and precision)
39+
40+
41+
## Example:
42+
3143
```sql title="Flink SQL"
3244

3345
CREATE TABLE VERSIONED (
@@ -52,23 +64,23 @@ SELECT * FROM VERSIONED;
5264

5365

5466
-- insert data with ts > 1000, update will be made
55-
INSERT INTO VERSIONED (a, b, ts) VALUES (1, 'v2', 2000);
67+
INSERT INTO VERSIONED (a, b, ts) VALUES (1, 'v3', 2000);
5668
SELECT * FROM VERSIONED;
5769
-- Output
5870
-- +---+-----+------+
5971
-- | a | b | ts |
6072
-- +---+-----+------+
61-
-- | 1 | v2 | 2000 |
73+
-- | 1 | v3 | 2000 |
6274
-- +---+-----+------+
6375

6476
-- insert data with ts = null, no update will be made
65-
INSERT INTO VERSIONED (a, b, ts) VALUES (1, 'v2', null);
77+
INSERT INTO VERSIONED (a, b, ts) VALUES (1, 'v4', null);
6678
SELECT * FROM VERSIONED;
6779
-- Output
6880
-- +---+-----+------+
6981
-- | a | b | ts |
7082
-- +---+-----+------+
71-
-- | 1 | v2 | 2000 |
83+
-- | 1 | v3 | 2000 |
7284
-- +---+-----+------+
7385

7486
```

0 commit comments

Comments
 (0)