Skip to content

Commit bd26ba1

Browse files
fanxishufanxishu
authored
Merge branch 'apache:dev' into dev
2 parents 41f8726 + eb70674 commit bd26ba1

File tree

423 files changed

+13520
-2970
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

423 files changed

+13520
-2970
lines changed

‎.github/workflows/backend.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1139,6 +1139,34 @@ jobs:
11391139
env:
11401140
MAVEN_OPTS: -Xmx4096m
11411141

1142+
jdbc-connectors-it-ddl:
1143+
needs: [ changes, sanity-check ]
1144+
if: needs.changes.outputs.api == 'true' || needs.changes.outputs.engine == 'true'
1145+
runs-on: ${{ matrix.os }}
1146+
env:
1147+
RUN_ALL_CONTAINER: ${{ needs.changes.outputs.api }}
1148+
RUN_ZETA_CONTAINER: ${{ needs.changes.outputs.engine }}
1149+
strategy:
1150+
matrix:
1151+
java: [ '8', '11' ]
1152+
os: [ 'ubuntu-latest' ]
1153+
timeout-minutes: 120
1154+
steps:
1155+
- uses: actions/checkout@v2
1156+
- name: Set up JDK ${{ matrix.java }}
1157+
uses: actions/setup-java@v3
1158+
with:
1159+
java-version: ${{ matrix.java }}
1160+
distribution: 'temurin'
1161+
cache: 'maven'
1162+
- name: free disk space
1163+
run: tools/github/free_disk_space.sh
1164+
- name: run jdbc connectors integration test (sink ddl)
1165+
run: |
1166+
./mvnw -B -T 1 verify -DskipUT=true -DskipIT=false -D"license.skipAddThirdParty"=true -D"skip.ui"=true --no-snapshot-updates -pl :connector-jdbc-e2e-ddl -am -Pci
1167+
env:
1168+
MAVEN_OPTS: -Xmx4096m
1169+
11421170
kudu-connector-it:
11431171
needs: [ changes, sanity-check ]
11441172
if: needs.changes.outputs.api == 'true' || contains(needs.changes.outputs.it-modules, 'connector-kudu-e2e')

‎bin/install-plugin.cmd‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ REM Get seatunnel home
2222
set "SEATUNNEL_HOME=%~dp0..\"
2323
echo Set SEATUNNEL_HOME to [%SEATUNNEL_HOME%]
2424

25-
REM Connector default version is 2.3.9, you can also choose a custom version. eg: 2.3.9: install-plugin.bat 2.3.9
26-
set "version=2.3.9"
25+
REM Connector default version is 2.3.10, you can also choose a custom version. eg: 2.3.10: install-plugin.bat 2.3.10
26+
set "version=2.3.10"
2727
if not "%~1"=="" set "version=%~1"
2828

2929
REM Create the lib directory

‎bin/install-plugin.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,8 +23,8 @@
2323
# get seatunnel home
2424
SEATUNNEL_HOME=$(cd $(dirname $0);cd ../;pwd)
2525

26-
# connector default version is 2.3.9, you can also choose a custom version. eg: 2.3.9: sh install-plugin.sh 2.3.9
27-
version=2.3.9
26+
# connector default version is 2.3.10, you can also choose a custom version. eg: 2.3.10: sh install-plugin.sh 2.3.10
27+
version=2.3.10
2828

2929
if [ -n "$1" ]; then
3030
version="$1"

‎deploy/kubernetes/seatunnel/Chart.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,8 @@ keywords:
3535
type: application
3636
# This is the chart version. This version number should be incremented each time you make changes
3737
# to the chart and its templates, including the app version.
38-
version: 2.3.9
38+
version: 2.3.10
3939

4040
# This is the version number of the application being deployed. This version number should be
4141
# incremented each time you make changes to the application.
42-
appVersion: 2.3.9
42+
appVersion: 2.3.10

‎docs/en/concept/schema-evolution.md‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,20 @@ Schema Evolution means that the schema of a data table can be changed and the da
2222
[Jdbc-Mysql](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md)
2323
[Jdbc-Oracle](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md)
2424
[Jdbc-Postgres](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md)
25+
[Jdbc-Dameng](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Jdbc.md)
2526
[StarRocks](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/StarRocks.md)
2627
[Doris](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Doris.md)
2728
[Paimon](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Paimon.md#Schema-Evolution)
2829
[Elasticsearch](https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Elasticsearch.md#Schema-Evolution)
2930

30-
Note: The schema evolution is not support the transform at now. The schema evolution of different types of databases（Oracle-CDC -> Jdbc-Mysql）is currently not supported the default value of the column in ddl.
31+
Note:
32+
* The schema evolution is not support the transform at now. The schema evolution of different types of databases（Oracle-CDC -> Jdbc-Mysql）is currently not supported the default value of the column in ddl.
3133

32-
When you use the Oracle-CDC，you can not use the username named `SYS` or `SYSTEM` to modify the table schema, otherwise the ddl event will be filtered out which can lead to the schema evolution not working.
34+
* When you use the Oracle-CDC，you can not use the username named `SYS` or `SYSTEM` to modify the table schema, otherwise the ddl event will be filtered out which can lead to the schema evolution not working.
3335
Otherwise, If your table name start with `ORA_TEMP_` will also has the same problem.
3436

37+
* Earlier versions of `Dameng` databases do not support the change of `Varchar` type fields to `Text` type fields.
38+
3539
## Enable schema evolution
3640
Schema evolution is disabled by default in CDC source. You need configure `schema-changes.enabled = true` which is only supported in CDC to enable it.
3741

@@ -287,4 +291,45 @@ sink {
287291
multi_table_sink_replica = 2
288292
}
289293
}
294+
```
295+
296+
### Mysql-CDC -> Jdbc-Dameng
297+
```hocon
298+
env {
299+
# You can set engine configuration here
300+
parallelism = 5
301+
job.mode = "STREAMING"
302+
checkpoint.interval = 5000
303+
read_limit.bytes_per_second=7000000
304+
read_limit.rows_per_second=400
305+
}
306+
307+
source {
308+
MySQL-CDC {
309+
server-id = 5652-5657
310+
username = "st_user_source"
311+
password = "mysqlpw"
312+
table-names = ["shop.products"]
313+
base-url = "jdbc:mysql://mysql_cdc_e2e:3306/shop"
314+
315+
schema-changes.enabled = true
316+
}
317+
}
318+
319+
sink {
320+
jdbc {
321+
url = "jdbc:dm://e2e_dmdb:5236"
322+
driver = "dm.jdbc.driver.DmDriver"
323+
connection_check_timeout_sec = 1000
324+
user = "SYSDBA"
325+
password = "SYSDBA"
326+
generate_sink_sql = true
327+
database = "DAMENG"
328+
table = "SYSDBA.sink_table_with_schema_change"
329+
primary_keys = ["id"]
330+
331+
# Validate ddl update for sink writer multi replica
332+
multi_table_sink_replica = 2
333+
}
334+
}
290335
```

‎docs/en/connector-v2/sink/Cassandra.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ Write data to Apache Cassandra.
1212

1313
## Options
1414

15-
| name | type | required | default value |
15+
| name | type | required | default value |
1616
|-------------------|---------|----------|---------------|
1717
| host | String | Yes | - |
1818
| keyspace | String | Yes | - |
@@ -21,7 +21,7 @@ Write data to Apache Cassandra.
2121
| password | String | No | - |
2222
| datacenter | String | No | datacenter1 |
2323
| consistency_level | String | No | LOCAL_ONE |
24-
| fields | String | No | LOCAL_ONE |
24+
| fields | Array | No | - |
2525
| batch_size | int | No | 5000 |
2626
| batch_type | String | No | UNLOGGED |
2727
| async_write | boolean | No | true |

‎docs/en/connector-v2/sink/CosFile.md‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ By default, we use 2PC commit to ensure `exactly-once`
5959
| common-options | object | no | - | |
6060
| max_rows_in_memory | int | no | - | Only used when file_format is excel. |
6161
| sheet_name | string | no | Sheet${Random number} | Only used when file_format is excel. |
62+
| csv_string_quote_mode | enum | no | MINIMAL | Only used when file_format is csv. |
6263
| xml_root_tag | string | no | RECORDS | Only used when file_format is xml. |
6364
| xml_row_tag | string | no | RECORD | Only used when file_format is xml. |
6465
| xml_use_attr_format | boolean | no | - | Only used when file_format is xml. |
@@ -107,7 +108,7 @@ Only used when `custom_filename` is `true`
107108

108109
When the format in the `file_name_expression` parameter is `xxxx-${now}` , `filename_time_format` can specify the time format of the path, and the default value is `yyyy.MM.dd` . The commonly used time formats are listed as follows:
109110

110-
| Symbol | Description |
111+
| Symbol | Description |
111112
|--------|--------------------|
112113
| y | Year |
113114
| M | Month |
@@ -199,6 +200,14 @@ When File Format is Excel,The maximum number of data items that can be cached in
199200

200201
Writer the sheet of the workbook
201202

203+
### csv_string_quote_mode [string]
204+
205+
When File Format is CSV,The string quote mode of CSV.
206+
207+
- ALL: All String fields will be quoted.
208+
- MINIMAL: Quotes fields which contain special characters such as a the field delimiter, quote character or any of the characters in the line separator string.
209+
- NONE: Never quotes fields. When the delimiter occurs in data, the printer prefixes it with the escape character. If the escape character is not set, format validation throws an exception.
210+
202211
### xml_root_tag [string]
203212

204213
Specifies the tag name of the root element within the XML file.

‎docs/en/connector-v2/sink/DB2.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ source {
116116
117117
transform {
118118
# If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
119-
# please go to https://seatunnel.apache.org/docs/category/transform-v2
119+
# please go to https://seatunnel.apache.org/docs/transform-v2
120120
}
121121
122122
sink {

‎docs/en/connector-v2/sink/Doris.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -49,17 +49,17 @@ The internal implementation of Doris sink connector is cached and imported by st
4949
| sink.label-prefix | String | Yes | - | The label prefix used by stream load imports. In the 2pc scenario, global uniqueness is required to ensure the EOS semantics of SeaTunnel. |
5050
| sink.enable-2pc | bool | No | false | Whether to enable two-phase commit (2pc), the default is false. For two-phase commit, please refer to [here](https://doris.apache.org/docs/data-operate/transaction?_highlight=two&_highlight=phase#stream-load-2pc). |
5151
| sink.enable-delete | bool | No | - | Whether to enable deletion. This option requires Doris table to enable batch delete function (0.15+ version is enabled by default), and only supports Unique model. you can get more detail at this [link](https://doris.apache.org/docs/dev/data-operate/delete/batch-delete-manual/) |
52-
| sink.check-interval | int | No | 10000 | check exception with the interval while loading |
53-
| sink.max-retries | int | No | 3 | the max retry times if writing records to database failed |
54-
| sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. |
55-
| sink.buffer-count | int | No | 3 | the buffer count to cache data for stream load. |
56-
| doris.batch.size | int | No | 1024 | the batch size of the write to doris each http request, when the row reaches the size or checkpoint is executed, the data of cached will write to server. |
57-
| needs_unsupported_type_casting | boolean | No | false | Whether to enable the unsupported type casting, such as Decimal64 to Double |
58-
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | the schema save mode, please refer to `schema_save_mode` below |
59-
| data_save_mode | Enum | no | APPEND_DATA | the data save mode, please refer to `data_save_mode` below |
60-
| save_mode_create_template | string | no | see below | see below |
61-
| custom_sql | String | no | - | When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks. |
62-
| doris.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. |
52+
| sink.check-interval | int | No | 10000 | check exception with the interval while loading |
53+
| sink.max-retries | int | No | 3 | the max retry times if writing records to database failed |
54+
| sink.buffer-size | int | No | 256 * 1024 | the buffer size to cache data for stream load. |
55+
| sink.buffer-count | int | No | 3 | the buffer count to cache data for stream load. |
56+
| doris.batch.size | int | No | 1024 | the batch size of the write to doris each http request, when the row reaches the size or checkpoint is executed, the data of cached will write to server. |
57+
| needs_unsupported_type_casting | boolean | No | false | Whether to enable the unsupported type casting, such as Decimal64 to Double |
58+
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | the schema save mode, please refer to `schema_save_mode` below |
59+
| data_save_mode | Enum | no | APPEND_DATA | the data save mode, please refer to `data_save_mode` below |
60+
| save_mode_create_template | string | no | see below | see below |
61+
| custom_sql | String | no | - | When data_save_mode selects CUSTOM_PROCESSING, you should fill in the CUSTOM_SQL parameter. This parameter usually fills in a SQL that can be executed. SQL will be executed before synchronization tasks. |
62+
| doris.config | map | yes | - | This option is used to support operations such as `insert`, `delete`, and `update` when automatically generate sql,and supported formats. |
6363

6464
### schema_save_mode[Enum]
6565

0 commit comments

Comments
 (0)