Skip to content

[Feature] Support flink 2.x version#636

Merged
JNSimba merged 23 commits into
apache:masterfrom
JNSimba:support-flink2.x
Feb 28, 2026
Merged

[Feature] Support flink 2.x version#636
JNSimba merged 23 commits into
apache:masterfrom
JNSimba:support-flink2.x

Conversation

@JNSimba
Copy link
Copy Markdown
Member

@JNSimba JNSimba commented Feb 27, 2026

Proposed changes

Because some APIs are incompatible between Flink 2.x and Flink 1.x, the project was refactored, resulting in the creation of the modules flink1 and flink2.

Flink1 supports Flink 1.15-1.20 and is compiled using JDK 1.8.

Flink2 supports Flink 2.0 and above and is compiled using JDK 17.

Checklist(Required)

  1. Does it affect the original behavior: (Yes/No/I Don't know)
  2. Has unit tests been added: (Yes/No/No Need)
  3. Has document been added or modified: (Yes/No/No Need)
  4. Does it need to update dependencies: (Yes/No)
  5. Are there any changes that cannot be rolled back: (Yes/No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@JNSimba JNSimba requested a review from Copilot February 27, 2026 09:22
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the Flink Doris Connector to support Flink 2.x by splitting version-specific integration into separate modules while keeping shared logic in a common base.

Changes:

  • Introduces Flink 1.x adapter wrappers around shared writer implementations (stream load / batch / copy).
  • Refactors core writers in base to be Flink-version-neutral (no direct Sink.InitContext dependency).
  • Updates build tooling, README, and CI workflows to build/test Flink 1.x (JDK 8) vs Flink 2.x (JDK 17).

Reviewed changes

Copilot reviewed 30 out of 394 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/sink/writer/DorisWriterAdapter.java Adds Flink 1.x wrapper delegating to version-neutral core DorisWriter.
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriterAdapter.java Adds Flink 1.x wrapper delegating to version-neutral core DorisCopyWriter.
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchWriterAdapter.java Adds Flink 1.x wrapper delegating to version-neutral core DorisBatchWriter.
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchSink.java Switches batch sink to use the new adapter.
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/sink/DorisSink.java Switches sink writer creation to use new adapters per write mode.
flink-doris-connector/flink-doris-connector-flink1/src/main/java/org/apache/doris/flink/catalog/DorisCatalog.java Formatting-only change to exception list.
flink-doris-connector/flink-doris-connector-flink1/pom.xml Introduces Flink 1.x module POM (Flink 1.20 / JDK 8).
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/sink/writer/TestRowDataSerializer.java Clears thread interrupt status in test setup.
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/sink/writer/TestDorisWriter.java Updates tests for refactored writer constructor and exception type.
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/sink/writer/TestDorisStreamLoad.java Clears thread interrupt status in test setup.
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/sink/copy/TestDorisCopyWriter.java Updates tests for refactored copy writer constructor.
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/sink/batch/TestDorisBatchWriter.java Updates tests for refactored batch writer constructor.
flink-doris-connector/flink-doris-connector-base/src/test/java/org/apache/doris/flink/container/AbstractContainerTestBase.java Ensures containers are closed via @AfterClass.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/table/DorisRowDataJdbcLookupFunction.java Adds eval(RowData) overload and delegates varargs eval to it.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/sink/writer/DorisWriter.java Makes core writer version-neutral by removing direct InitContext dependency.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/sink/copy/DorisCopyWriter.java Makes core copy writer version-neutral by removing direct InitContext dependency.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/sink/copy/BatchRecordBuffer.java Javadoc formatting improvements.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/sink/batch/DorisBatchWriter.java Makes core batch writer version-neutral by removing direct InitContext dependency.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/sink/batch/BatchRecordBuffer.java Javadoc formatting improvements.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/cfg/DorisReadOptions.java Minor formatting update.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/cfg/DorisConnectionOptions.java Minor formatting update.
flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/catalog/DorisTypeMapper.java Minor formatting update.
flink-doris-connector/flink-doris-connector-base/pom.xml Introduces base module POM for shared code built per profile.
flink-doris-connector/build.sh Adds Flink 2.x options, enforces JDK version, builds module/profile-specific artifacts.
README.md Documents supported Flink versions and JDK requirements.
.github/workflows/run-itcase-flink2.yml Adds ITCase workflow for Flink 2.x on JDK 17.
.github/workflows/run-itcase-flink1.yml Renames/adjusts ITCase workflow to be Flink 1.x specific.
.github/workflows/run-e2ecase-flink2.yml Adds E2E workflow for Flink 2.x on JDK 17.
.github/workflows/run-e2ecase-flink1.yml Renames/adjusts E2E workflow to be Flink 1.x specific.
.github/workflows/build-connector.yml Builds both Flink 1.x (JDK 8) and Flink 2.x (JDK 17) connector variants.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread flink-doris-connector/build.sh
Comment thread flink-doris-connector/build.sh
Comment thread .github/workflows/run-itcase-flink2.yml
@JNSimba JNSimba mentioned this pull request Feb 28, 2026
@JNSimba JNSimba merged commit f766f1f into apache:master Feb 28, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants