Skip to content

Commit f766f1f

Browse files
JNSimbaJNSimba
authored
[Feature] Support flink 2.x version (#636)
Because some APIs are incompatible between Flink 2.x and Flink 1.x, the project was refactored, resulting in the creation of the modules flink1 and flink2. Flink1 supports Flink 1.15-1.20 and is compiled using JDK 1.8. Flink2 supports Flink 2.0 and above and is compiled using JDK 17.
1 parent e10a47c commit f766f1f

396 files changed

Lines changed: 5858 additions & 696 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/build-connector.yml‎

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -41,41 +41,78 @@ jobs:
4141
- name: Build flink connector 1.15
4242
run: |
4343
cd flink-doris-connector && mvn clean package \
44+
-pl flink-doris-connector-flink1 -am \
45+
-Pflink1 \
4446
-Dflink.version=1.15.0 \
45-
-Dflink.minor.version=1.15 \
47+
-Dflink.major.version=1.15 \
4648
-Dflink.python.id=flink-python_2.12
4749
4850
- name: Build flink connector 1.16
4951
run: |
5052
cd flink-doris-connector && mvn clean package \
53+
-pl flink-doris-connector-flink1 -am \
54+
-Pflink1 \
5155
-Dflink.version=1.16.0 \
52-
-Dflink.minor.version=1.16 \
53-
-Dflink.python.id=flink-python
56+
-Dflink.major.version=1.16
5457
5558
- name: Build flink connector 1.17
5659
run: |
5760
cd flink-doris-connector && mvn clean package \
61+
-pl flink-doris-connector-flink1 -am \
62+
-Pflink1 \
5863
-Dflink.version=1.17.0 \
59-
-Dflink.minor.version=1.17 \
60-
-Dflink.python.id=flink-python
64+
-Dflink.major.version=1.17
6165
6266
- name: Build flink connector 1.18
6367
run: |
6468
cd flink-doris-connector && mvn clean package \
69+
-pl flink-doris-connector-flink1 -am \
70+
-Pflink1 \
6571
-Dflink.version=1.18.0 \
66-
-Dflink.minor.version=1.18 \
67-
-Dflink.python.id=flink-python
72+
-Dflink.major.version=1.18
6873
6974
- name: Build flink connector 1.19
7075
run: |
7176
cd flink-doris-connector && mvn clean package \
77+
-pl flink-doris-connector-flink1 -am \
78+
-Pflink1 \
7279
-Dflink.version=1.19.0 \
73-
-Dflink.minor.version=1.19 \
74-
-Dflink.python.id=flink-python
80+
-Dflink.major.version=1.19
7581
7682
- name: Build flink connector 1.20
7783
run: |
7884
cd flink-doris-connector && mvn clean package \
85+
-pl flink-doris-connector-flink1 -am \
86+
-Pflink1 \
7987
-Dflink.version=1.20.0 \
80-
-Dflink.minor.version=1.20 \
81-
-Dflink.python.id=flink-python
88+
-Dflink.major.version=1.20
89+
90+
- name: Setup java
91+
uses: actions/setup-java@v2
92+
with:
93+
distribution: adopt
94+
java-version: '17'
95+
96+
- name: Build flink connector 2.0
97+
run: |
98+
cd flink-doris-connector && mvn clean package \
99+
-pl flink-doris-connector-flink2 -am \
100+
-Pflink2 \
101+
-Dflink.version=2.0.0 \
102+
-Dflink.major.version=2.0
103+
104+
- name: Build flink connector 2.1
105+
run: |
106+
cd flink-doris-connector && mvn clean package \
107+
-pl flink-doris-connector-flink2 -am \
108+
-Pflink2 \
109+
-Dflink.version=2.1.0 \
110+
-Dflink.major.version=2.1
111+
112+
- name: Build flink connector 2.2
113+
run: |
114+
cd flink-doris-connector && mvn clean package \
115+
-pl flink-doris-connector-flink2 -am \
116+
-Pflink2 \
117+
-Dflink.version=2.2.0 \
118+
-Dflink.major.version=2.2

‎.github/workflows/checkstyle.yaml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,22 @@ jobs:
3232
persist-credentials: false
3333
submodules: recursive
3434

35-
- name: Setup java
35+
- name: Setup java 8
3636
uses: actions/setup-java@v2
3737
with:
3838
distribution: adopt
3939
java-version: '8'
4040

4141
- name: Run java checkstyle
4242
run:
43-
cd flink-doris-connector && mvn clean compile checkstyle:checkstyle
43+
cd flink-doris-connector && mvn clean checkstyle:check -Pflink1 -pl flink-doris-connector-flink1 -am
44+
45+
- name: Setup java 17
46+
uses: actions/setup-java@v2
47+
with:
48+
distribution: adopt
49+
java-version: '17'
50+
51+
- name: Run java checkstyle
52+
run:
53+
cd flink-doris-connector && mvn clean checkstyle:check -Pflink2 -pl flink-doris-connector-flink2 -am

.github/workflows/run-e2ecase.yml renamed to .github/workflows/run-e2ecase-flink1.yml

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
# under the License.
1717
#
1818
---
19-
name: Run E2ECases
19+
name: Run E2ECases Flink1.x
2020
on:
2121
pull_request:
2222
push:
@@ -38,7 +38,10 @@ jobs:
3838
distribution: adopt
3939
java-version: '8'
4040

41-
- name: Run E2ECases
41+
- name: Run E2ECases (Flink1 module)
4242
run: |
43-
cd flink-doris-connector && mvn test -Dtest="*E2ECase" -Dimage="apache/doris:doris-all-in-one-2.1.0"
44-
43+
cd flink-doris-connector && mvn test \
44+
-Pflink1 -pl flink-doris-connector-it -am \
45+
-DfailIfNoTests=false \
46+
-Dtest="*E2ECase" \
47+
-Dimage="apache/doris:doris-all-in-one-2.1.0"

‎.github/workflows/run-e2ecase-flink2.yml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
---
19+
name: Run E2ECases Flink2.x
20+
on:
21+
pull_request:
22+
push:
23+
24+
jobs:
25+
build-extension:
26+
name: "Run E2ECases"
27+
runs-on: ubuntu-latest
28+
defaults:
29+
run:
30+
shell: bash
31+
steps:
32+
- name: Checkout
33+
uses: actions/checkout@master
34+
35+
- name: Setup java
36+
uses: actions/setup-java@v2
37+
with:
38+
distribution: adopt
39+
java-version: '17'
40+
41+
- name: Run E2ECases (Flink2 module)
42+
run: |
43+
cd flink-doris-connector && mvn test \
44+
-Pflink2 -pl flink-doris-connector-it -am \
45+
-DfailIfNoTests=false \
46+
-Dtest="*E2ECase" \
47+
-Dimage="apache/doris:doris-all-in-one-2.1.0"

.github/workflows/run-itcase.yml renamed to .github/workflows/run-itcase-flink1.yml

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
# under the License.
1717
#
1818
---
19-
name: Run ITCases
19+
name: Run ITCases Flink1.x
2020
on:
2121
pull_request:
2222
push:
@@ -38,7 +38,10 @@ jobs:
3838
distribution: adopt
3939
java-version: '8'
4040

41-
- name: Run ITCases
41+
- name: Run ITCases (Flink1 module)
4242
run: |
43-
cd flink-doris-connector && mvn test -Dtest="*ITCase" -Dimage="apache/doris:doris-all-in-one-2.1.0"
44-
43+
cd flink-doris-connector && mvn test \
44+
-Pflink1 -pl flink-doris-connector-it -am \
45+
-DfailIfNoTests=false \
46+
-Dtest="*ITCase" \
47+
-Dimage="apache/doris:doris-all-in-one-2.1.0"

‎.github/workflows/run-itcase-flink2.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
#
18+
---
19+
name: Run ITCases Flink2.x
20+
on:
21+
pull_request:
22+
push:
23+
24+
jobs:
25+
build-extension:
26+
name: "Run ITCases"
27+
runs-on: ubuntu-latest
28+
defaults:
29+
run:
30+
shell: bash
31+
steps:
32+
- name: Checkout
33+
uses: actions/checkout@master
34+
35+
- name: Setup java
36+
uses: actions/setup-java@v2
37+
with:
38+
distribution: adopt
39+
java-version: '17'
40+
41+
- name: Run ITCases (Flink2 module)
42+
run: |
43+
cd flink-doris-connector && mvn test \
44+
-Pflink2 -pl flink-doris-connector-it -am \
45+
-DfailIfNoTests=false \
46+
-Dtest="*ITCase" \
47+
-Dimage="apache/doris:doris-all-in-one-2.1.0"
48+

‎.licenserc.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,6 @@ header:
1212
- '.github/PULL_REQUEST_TEMPLATE.md'
1313
- '.licenserc.yaml'
1414
- 'custom_env.sh.tpl'
15-
- 'flink-doris-connector/src/test/resources/container/'
15+
- 'flink-doris-connector/flink-doris-connector-base/src/test/resources/container/'
1616

1717
comment: on-failure

‎README.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,19 @@ under the License.
2424

2525
## Flink Doris Connector
2626

27-
Flink Doris Connector now support flink version from 1.11 to 1.20.
27+
Flink Doris Connector supports the following Flink versions:
28+
29+
| Flink Version | JDK Requirement |
30+
|:--------------|:----------------|
31+
| 1.15.x | JDK 8 |
32+
| 1.16.x | JDK 8 |
33+
| 1.17.x | JDK 8 |
34+
| 1.18.x | JDK 8 |
35+
| 1.19.x | JDK 8 |
36+
| 1.20.x | JDK 8 |
37+
| 2.0.x | JDK 17 |
38+
| 2.1.x | JDK 17 |
39+
| 2.2.x | JDK 17 |
2840

2941
If you wish to contribute or use a connector from flink 1.13 (and earlier), please use the [branch-for-flink-before-1.13](https://github.com/apache/doris-flink-connector/tree/branch-for-flink-before-1.13)
3042

@@ -37,6 +49,9 @@ More information about compilation and usage, please visit [Flink Doris Connecto
3749
## How to Build
3850

3951
You need to copy customer_env.sh.tpl to customer_env.sh before build and you need to configure it before build.
52+
53+
> **JDK requirement:** Use JDK 8 for Flink 1.x, and JDK 17 for Flink 2.x.
54+
4055
```shell
4156
git clone git@github.com:apache/doris-flink-connector.git
4257
cd doris-flink-connector/flink-doris-connector

0 commit comments

Comments
 (0)