Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,66 @@ sidebar_position: 3

# Versioned Merge Engine

TODO: Fill me #459
By specifying `'table.merge-engine' = 'versioned'`, users can update data based on the configured version column. Updates will be carried out when the latest value of the specified field is greater than the stored value. If it is less than or empty, no update will be made.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
By specifying `'table.merge-engine' = 'versioned'`, users can update data based on the configured version column. Updates will be carried out when the latest value of the specified field is greater than the stored value. If it is less than or empty, no update will be made.
By setting `'table.merge-engine' = 'versioned'`, users can update data based on the configured version column. Updates will be carried out when the latest value of the specified field is greater than or equal to the stored value. If it is less than or null, no update will be made.


Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to suggest to add a statement to explain how the feature is useful just like first-row merge page does, maybe something like:

This feature is particularly valuable for replacing [Deduplication](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/dev/table/sql/queries/deduplication/) transformations in streaming computations, reducing complexity and improving overall efficiency.

:::note
When using `versioned` merge engine, there are the following limits:
- `UPDATE` and `DELETE` statements are not supported
- Partial update is not supported
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add one more note just like first-row merge engine:

`UPDATE_BEFORE` and `DELETE` changelog events are ignored automatically

:::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:::
:::


## Versioned Merge Column

:::note
The versioned merge column supports the following data types.
- INT
- BIGINT
- TIMESTAMP
- TIMESTAMP(p)
- TIMESTAMP_LTZ
- TIMESTAMP_LTZ(p)
:::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
:::
:::


example:
```sql title="versioned"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
```sql title="versioned"
```sql title="Flink SQL"


create table merge_engine_with_version (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please change SQL keyword to uppercase to align the style of first-row merge engine page?

a int not null primary key not enforced,
b string,
ts bigint
) with(
'table.merge-engine' = 'versioned',
'table.merge-engine.versioned.ver-column' = 'ts'
);
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v1', 1000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v1', 1000);
insert into merge_engine_with_version (a, b, ts ) VALUES (1, 'v1', 1000);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@luoyuxia Whether to remove the space after ts?
(a, b, ts ) -> (a, b, ts)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't noticed that


-- insert data with ts < 1000, no update will be made
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', 999);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', 999);
insert into merge_engine_with_version (a, b, ts ) VALUES (1, 'v2', 999);

select * from merge_engine_with_version;
-- Output
-- +---+-----+------+
-- | a | b | ts |
-- +---+-----+------+
-- | 1 | v1 | 1000 |
-- +---+-----+------+


-- insert data with ts > 1000, update will be made
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', 2000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit

Suggested change
insert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', 2000);
insert into merge_engine_with_version (a, b, ts ) VALUES (1, 'v2', 2000);

select * from merge_engine_with_version;
-- Output
-- +---+-----+------+
-- | a | b | ts |
-- +---+-----+------+
-- | 1 | v2 | 2000 |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add -- +---+-----+------+ in table end.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: add the bottom line to align with the style of above content

-- +---+-----+------+
-- | a | b   | ts   |
-- +---+-----+------+
-- | 1 | v2  | 2000 |
-- +---+-----+------+

-- insert data with ts = null, no update will be made
nsert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', null);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: nsert -> insert.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nsert into merge_engine_with_version ( a,b,ts ) VALUES (1, 'v2', null);
insert into merge_engine_with_version (a, b, ts ) VALUES (1, 'v2', null);

select * from merge_engine_with_version;
-- Output
-- +---+-----+------+
-- | a | b | ts |
-- +---+-----+------+
-- | 1 | v2 | 2000 |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dito

```