Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit b0abbd2

Browse files
authoredMar 19, 2024··
[Feature][Connector-V2] Support multi-table sink feature for paimon #5652 (#6449)
1 parent cb3db75 commit b0abbd2

File tree

27 files changed

+1652
-128
lines changed

27 files changed

+1652
-128
lines changed
 

Diff for: ‎docs/en/connector-v2/sink/Paimon.md

+59-23
Original file line numberDiff line numberDiff line change
@@ -4,48 +4,84 @@
44
55
## Description
66

7-
Write data to Apache Paimon.
7+
Sink connector for Apache Paimon. It can support cdc mode 、auto create table.
88

99
## Key features
1010

1111
- [x] [exactly-once](../../concept/connector-v2-features.md)
1212

1313
## Options
1414

15-
| name | type | required | default value |
16-
|----------------|--------|----------|---------------|
17-
| warehouse | String | Yes | - |
18-
| database | String | Yes | - |
19-
| table | String | Yes | - |
20-
| hdfs_site_path | String | No | - |
15+
| name | type | required | default value | Description |
16+
|------------------|--------|----------|------------------------------|---------------------------------|
17+
| warehouse | String | Yes | - | Paimon warehouse path |
18+
| database | String | Yes | - | The database you want to access |
19+
| table | String | Yes | - | The table you want to access |
20+
| hdfs_site_path | String | No | - | |
21+
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | The schema save mode |
22+
| data_save_mode | Enum | no | APPEND_DATA | The data save mode |
2123

22-
### warehouse [string]
23-
24-
Paimon warehouse path
25-
26-
### database [string]
24+
## Examples
2725

28-
The database you want to access
26+
### Single table
2927

30-
### table [String]
28+
```hocon
29+
env {
30+
parallelism = 1
31+
job.mode = "STREAMING"
32+
checkpoint.interval = 5000
33+
}
3134
32-
The table you want to access
35+
source {
36+
Mysql-CDC {
37+
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
38+
username = "root"
39+
password = "******"
40+
table-names = ["seatunnel.role"]
41+
}
42+
}
3343
34-
## Examples
44+
transform {
45+
}
3546
36-
```hocon
3747
sink {
3848
Paimon {
39-
warehouse = "/tmp/paimon"
40-
database = "default"
41-
table = "st_test"
49+
catalog_name="seatunnel_test"
50+
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
51+
database="seatunnel"
52+
table="role"
4253
}
4354
}
4455
```
4556

46-
## Changelog
57+
### Multiple table
58+
59+
```hocon
60+
env {
61+
parallelism = 1
62+
job.mode = "STREAMING"
63+
checkpoint.interval = 5000
64+
}
65+
66+
source {
67+
Mysql-CDC {
68+
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
69+
username = "root"
70+
password = "******"
71+
table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
72+
}
73+
}
4774
48-
### next version
75+
transform {
76+
}
4977
50-
- Add Paimon Sink Connector
78+
sink {
79+
Paimon {
80+
catalog_name="seatunnel_test"
81+
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
82+
database="${database_name}"
83+
table="${table_name}"
84+
}
85+
}
86+
```
5187

Diff for: ‎docs/zh/connector-v2/sink/Paimon.md

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Paimon
2+
3+
> Paimon 数据连接器
4+
5+
## 描述
6+
7+
Apache Paimon数据连接器。支持cdc写以及自动建表。
8+
9+
## 主要特性
10+
11+
- [x] [exactly-once](../../concept/connector-v2-features.md)
12+
13+
## 连接器选项
14+
15+
| 名称 | 类型 | 是否必须 | 默认值 | 描述 |
16+
|------------------|--------|------|------------------------------|--------------------|
17+
| warehouse | String | Yes | - | Paimon warehouse路径 |
18+
| database | String | Yes | - | 数据库名称 |
19+
| table | String | Yes | - | 表名 |
20+
| hdfs_site_path | String | No | - | |
21+
| schema_save_mode | Enum | no | CREATE_SCHEMA_WHEN_NOT_EXIST | schema保存模式 |
22+
| data_save_mode | Enum | no | APPEND_DATA | 数据保存模式 |
23+
24+
## 示例
25+
26+
### 单表
27+
28+
```hocon
29+
env {
30+
parallelism = 1
31+
job.mode = "STREAMING"
32+
checkpoint.interval = 5000
33+
}
34+
35+
source {
36+
Mysql-CDC {
37+
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
38+
username = "root"
39+
password = "******"
40+
table-names = ["seatunnel.role"]
41+
}
42+
}
43+
44+
transform {
45+
}
46+
47+
sink {
48+
Paimon {
49+
catalog_name="seatunnel_test"
50+
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
51+
database="seatunnel"
52+
table="role"
53+
}
54+
}
55+
```
56+
57+
### 多表
58+
59+
```hocon
60+
env {
61+
parallelism = 1
62+
job.mode = "STREAMING"
63+
checkpoint.interval = 5000
64+
}
65+
66+
source {
67+
Mysql-CDC {
68+
base-url = "jdbc:mysql://127.0.0.1:3306/seatunnel"
69+
username = "root"
70+
password = "******"
71+
table-names = ["seatunnel.role","seatunnel.user","galileo.Bucket"]
72+
}
73+
}
74+
75+
transform {
76+
}
77+
78+
sink {
79+
Paimon {
80+
catalog_name="seatunnel_test"
81+
warehouse="file:///tmp/seatunnel/paimon/hadoop-sink/"
82+
database="${database_name}"
83+
table="${table_name}"
84+
}
85+
}
86+
```
87+

Diff for: ‎seatunnel-connectors-v2/connector-paimon/pom.xml

+5
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,11 @@
3434
</properties>
3535

3636
<dependencies>
37+
<dependency>
38+
<groupId>org.apache.seatunnel</groupId>
39+
<artifactId>connector-common</artifactId>
40+
<version>${project.version}</version>
41+
</dependency>
3742

3843
<dependency>
3944
<groupId>org.apache.paimon</groupId>

0 commit comments

Comments
 (0)