Skip to content

Conversation

@xuyangzhong
Copy link
Contributor

Purpose

Linked issue: close #1739

Brief change log

Add docs about delta join support with Flink 2.1

Tests

Run npm run start to check.

API and Format

None

Documentation

None


The work on Delta Join is still ongoing, so the support for more sql patterns that can be optimized into delta join varies across different versions of Flink. More details can be found at [Delta Join](https://issues.apache.org/jira/browse/FLINK-37836).

### Flink 2.1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the upcoming Flink 2.2, additional patterns will be supported, as the relevant PRs have already been merged. I'm uncertain whether I should include a description of Flink 2.2 here. So it's entirely up to you.

### Flink 2.2 (upcoming)

#### Supported Features

- Support for optimizing a dual-stream join from CDC sources that do not include delete messages into a delta join.
  - Include the `table.delete.behavior` with a non-`ALLOW` option in the source table DDL to ensure that the source table does not emit delete messages.
- Support `Project` and `Filter` between source and delta join.
- Support cache in delta join.

#### Limitations

- The primary key or the prefix lookup key of the tables must be included as part of the equivalence conditions in the join.
- The join must be a INNER join.
- The downstream nodes of the join can accept duplicate changes, such as a sink that provides UPSERT mode.
- When consuming a CDC stream, the join key used in the delta join must be part of the primary key.
- All filters must be applied on the upsert key, and neither filters nor projections should contain non-deterministic functions.

---
title: "DataStream API"
sidebar_position: 6
sidebar_position: 7
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the position here to set DeltaJoins page between Lookups and DataStream API .
image

@polyzos
Copy link
Contributor

polyzos commented Oct 29, 2025

@xuyangzhong i added a few improvements.. please take a look to verify and let me know if you are ok with these
(i wanted to add a section to explain what prefix keys are)

@xuyangzhong
Copy link
Contributor Author

@polyzos Thanks for reviewing. I just do a few minor adjustments on your great improvements. By the way, I think you're right we maybe need a new page to describe prefix keys. Prior to this, only the Lookups page had a brief description: https://fluss.apache.org/docs/0.8/engine-flink/lookups/#prefix-lookup.

Copy link
Member

@wuchong wuchong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I improved the delta join documentation a bit, including

  • Add a image to explain the delta join
  • Understanding Prefix Keys -> Understanding Index Keys as the index key is more generic, Prefix Key is just an implementation of Index Key, and we are supporting the general index key in the next version.
  • Add "Future Plan"

@wuchong wuchong merged commit aa4afe8 into apache:main Nov 2, 2025
2 checks passed
wuchong pushed a commit that referenced this pull request Nov 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add docs for delta join support with Flink 2.1

3 participants