Skip to content

[Improve][Connector-V2] Optimize dialect selection in jdbc #8820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 2, 2025
Merged

[Improve][Connector-V2] Optimize dialect selection in jdbc #8820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
dd51e9d
[Improve][Connector-V2] Optimize dialect selection in jdbc
corgy-w Feb 25, 2025
a8bcd66
[Improve][Connector-V2] Optimize doc
corgy-w Feb 26, 2025
88ec7f1
[Improve][Connector-V2] Optimize code
corgy-w Feb 26, 2025
d902023
[Improve][Connector-V2] Optimize doc
corgy-w Mar 11, 2025
9c6ac08
[Fix][Connector-V2] opt
corgy-w Mar 18, 2025
f4e3b4e
[Fix][Connector-V2] opt
corgy-w Mar 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion docs/en/connector-v2/sink/Jdbc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,14 +31,15 @@ support `Xa transactions`. You can set `is_exactly_once=true` to enable it.

## Options

| Name | Type | Required | Default |
| Name | Type | Required | Default |
|-------------------------------------------|---------|----------|------------------------------|
| url | String | Yes | - |
| driver | String | Yes | - |
| user | String | No | - |
| password | String | No | - |
| query | String | No | - |
| compatible_mode | String | No | - |
| dialect | String | No | - |
| database | String | No | - |
| table | String | No | - |
| primary_keys | Array | No | - |
Expand Down Expand Up @@ -90,6 +91,23 @@ For example, when using OceanBase database, you need to set it to 'mysql' or 'or

Postgres 9.5 version or below,please set it to `postgresLow` to support cdc

### dialect [string]

The appointed dialect, if it does not exist, is still obtained according to the url, and the priority is higher than the url. For example,when using starrocks, you need set it to `starrocks`. Similarly, when using mysql, you need to set its value to `mysql`.

#### dialect list

| | Dialect Name | |
|-----------|--------------|----------|
| Greenplum | DB2 | Dameng |
| Gbase8a | HIVE | KingBase |
| MySQL | StarRocks | Oracle |
| Phoenix | Postgres | Redshift |
| SapHana | Snowflake | Sqlite |
| SqlServer | Tablestore | Teradata |
| Vertica | OceanBase | XUGU |
| IRIS | Inceptor | Highgo |

### database [string]

Use this `database` and `table-name` auto-generate sql and receive upstream input datas write to database.
Expand All @@ -105,11 +123,13 @@ This option is mutually exclusive with `query` and has a higher priority.
The table parameter can fill in the name of an unwilling table, which will eventually be used as the table name of the creation table, and supports variables (`${table_name}`, `${schema_name}`). Replacement rules: `${schema_name}` will replace the SCHEMA name passed to the target side, and `${table_name}` will replace the name of the table passed to the table at the target side.

mysql sink for example:

1. test_${schema_name}_${table_name}_test
2. sink_sinktable
3. ss_${table_name}

pgsql (Oracle Sqlserver ...) Sink for example:

1. ${schema_name}.${table_name} _test
2. dbo.tt_${table_name} _sink
3. public.sink_table
Expand Down
19 changes: 19 additions & 0 deletions docs/en/connector-v2/source/Jdbc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ supports query SQL and can achieve projection effect.
| password | String | No | - | password |
| query | String | No | - | Query statement |
| compatible_mode | String | No | - | The compatible mode of database, required when the database supports multiple compatible modes.<br/> For example, when using OceanBase database, you need to set it to 'mysql' or 'oracle'. <br/> when using starrocks, you need set it to `starrocks` |
| dialect | String | No | - | The appointed dialect, if it does not exist, is still obtained according to the url, and the priority is higher than the url. <br/> For example,when using starrocks, you need set it to `starrocks` |
Copy link
Preview

Copilot AI Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] There is a missing space after the comma in 'For example,when using starrocks,'.

Suggested change
| dialect | String | No | - | The appointed dialect, if it does not exist, is still obtained according to the url, and the priority is higher than the url. <br/> For example,when using starrocks, you need set it to `starrocks` |
| dialect | String | No | - | The appointed dialect, if it does not exist, is still obtained according to the url, and the priority is higher than the url. <br/> For example, when using starrocks, you need set it to `starrocks` |

Copilot uses AI. Check for mistakes.

| connection_check_timeout_sec | Int | No | 30 | The time in seconds to wait for the database operation used to validate the connection to complete. |
| partition_column | String | No | - | The column name for split data. |
| partition_upper_bound | Long | No | - | The partition_column max value for scan, if not set SeaTunnel will query database get max value. |
Expand Down Expand Up @@ -93,6 +94,24 @@ decimal_type_narrowing = false
| NUMBER(6, 0) | Decimal(6, 0) |
| NUMBER(10, 0) | Decimal(10, 0) |

### dialect [string]

The appointed dialect, if it does not exist, is still obtained according to the url, and the priority is higher than the url. For example,when using starrocks, you need set it to `starrocks`. Similarly, when using mysql, you need to set its value to `mysql`.

#### dialect list

| | Dialect Name | |
|-----------|--------------|----------|
| Greenplum | DB2 | Dameng |
| Gbase8a | HIVE | KingBase |
| MySQL | StarRocks | Oracle |
| Phoenix | Postgres | Redshift |
| SapHana | Snowflake | Sqlite |
| SqlServer | Tablestore | Teradata |
| Vertica | OceanBase | XUGU |
| IRIS | Inceptor | Highgo |


## Parallel Reader

The JDBC Source connector supports parallel reading of data from tables. SeaTunnel will use certain rules to split the data in the table, which will be handed over to readers for reading. The number of readers is determined by the `parallelism` option.
Expand Down
Loading
Loading