Skip to content

[FLINK-36796][pipeline-connector][oracle]add oracle pipeline connector.#3995

Merged
lvyanquan merged 7 commits into
apache:masterfrom
linjianchang:master-36796
Feb 10, 2026
Merged

[FLINK-36796][pipeline-connector][oracle]add oracle pipeline connector.#3995
lvyanquan merged 7 commits into
apache:masterfrom
linjianchang:master-36796

Conversation

@linjianchang
Copy link
Copy Markdown
Contributor

add oracle pipeline connector.

Copy link
Copy Markdown
Contributor

@joyCurry30 joyCurry30 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution. I just left some comment.

<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-core</artifactId>
<version>1.9.8.Final</version>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use ${debezium.version}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the scope should be "provide".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been modified

<dependency>
<groupId>com.ververica</groupId>
<artifactId>flink-cdc-source-e2e-tests</artifactId>
<version>cty-3.0-2.2-SNAPSHOT</version>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why you dependent on "flink-cdc-source-e2e-tests"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been removed

<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-test-util</artifactId>
<version>3.4-SNAPSHOT</version>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use "${project.version}".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been modified

<include>io.debezium:debezium-core</include>
<include>io.debezium:debezium-ddl-parser</include>
<include>io.debezium:debezium-connector-oracle</include>
<include>io.debezium:debezium-connector-mysql</include>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain the rationale behind having a dependency on the MySQL CDC connector within the Oracle CDC connector implementation? I'd like to better understand how these components interact in this context.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been removed

<include>io.debezium:debezium-connector-oracle</include>
<include>io.debezium:debezium-connector-mysql</include>
<include>com.ververica:flink-connector-debezium</include>
<include>com.ververica:flink-connector-mysql-cdc</include>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been removed

<include>org.antlr:antlr4-runtime</include>
<include>org.apache.kafka:*</include>
<include>mysql:mysql-connector-java</include>
<include>com.zendesk:mysql-binlog-connector-java</include>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been removed

Comment on lines +34 to +38
public static final ConfigOption<String> JDBC_URL =
ConfigOptions.key("jdbc.url")
.stringType()
.noDefaultValue()
.withDescription("The jdbc url.");
Copy link
Copy Markdown
Contributor

@joyCurry30 joyCurry30 Apr 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we clarify the relationship between the JDBC URL and the individual hostname/port parameters?

If we already have a “jdbc.url” configuration field, is there still value in maintaining separate “hostname and “port” parameters? Should these be mutually exclusive?

When using “hostname” and "port" configuration, is there a parameter to explicitly specify the driver type (Thin vs OCI)? This would be crucial for constructing the correct JDBC connection string format.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In The original oracle module [flink-connector-oracle-cdc]OracleJdbcUrlUtils#getConnectionUrlWithSid method,url,hostname and port is not exclusive,when url config is null,url = "jdbc:oracle:thin:@" + hostname + ":" + port + ":" + dbname;,but 19c url is must connect through the service name,like url = "jdbc:oracle:thin:@" + hostname + ":" + port + "/" + dbname; So you can configure the URL to adapt to oracle 19c

Comment on lines +140 to +143
if (isAddMeta) {
map.put(OracleDataSourceOptions.HOSTNAME.key(), hostname);
map.put(OracleDataSourceOptions.PORT.key(), port);
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify the design intent behind treating hostname and port as common metadata fields?

Since we don't currently support multi-source CDC synchronization, are these fields intended for future extensibility?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have been removed

@github-actions github-actions Bot added the docs Improvements or additions to documentation label Apr 21, 2025
@linjianchang
Copy link
Copy Markdown
Contributor Author

@joyCurry30 Already modified,please review again ,thanks!

Copy link
Copy Markdown
Contributor

@joyCurry30 joyCurry30 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thank you for your contribution. I left some comments for the doc.


```yaml
source:
type: mysql
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“type” should be "oracle".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +172 to +173
- `initial` (默认):在第一次启动时对受监视的数据库表执行初始快照,并继续读取最新的 binlog。
- `latest-offset`:首次启动时,从不对受监视的数据库表执行快照, 连接器仅从 binlog 的结尾处开始读取,这意味着连接器只能读取在连接器启动之后的数据更改。
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does oracle cdc use binlog?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +261 to +278
<tr>
<td>
XMLTYPE
</td>
<td>VARCHAR(n)</td>
<td></td>
</tr>
<tr>
<td>
VARCHAR(n)<br>
VARCHAR2(n)<br>
NVARCHAR2(n)<br>
NCHAR(n)<br>
CHAR(n)<br>
</td>
<td>VARCHAR(n)</td>
<td></td>
</tr>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two elements should be merged.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified


```yaml
source:
type: mysql
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +176 to +178
- `initial` (default): Performs an initial snapshot on the monitored database tables upon first startup, and continue to read the latest binlog.
- `latest-offset`: Never to perform snapshot on the monitored database tables upon first startup, just read from

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +189 to +190
## 数据类型映射

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use English, please.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment thread docs/content/docs/connectors/pipeline-connectors/oracle.md
Comment on lines +86 to +99
.hostname(
config.getOptional(OracleDataSourceOptions.HOSTNAME).get())
.port(config.getOptional(OracleDataSourceOptions.PORT).get())
.databaseList(
config.getOptional(OracleDataSourceOptions.DATABASE)
.get()) // monitor oracledatabase
.tableList(
config.getOptional(OracleDataSourceOptions.TABLES)
.get()) // monitor productstable
.username(
config.getOptional(OracleDataSourceOptions.USERNAME).get())
.password(
config.getOptional(OracleDataSourceOptions.PASSWORD).get())
.includeSchemaChanges(true);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use config.get(ConfigOption option).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

options.add(OracleDataSourceOptions.SCHEMALIST);
options.add(OracleDataSourceOptions.DATABASE);
options.add(OracleDataSourceOptions.TABLES);
options.add(METADATA_LIST);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use OracleDataSourceOptions.METADATA_LIST.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +194 to +198
private static final String SCAN_STARTUP_MODE_VALUE_INITIAL = "initial";
private static final String SCAN_STARTUP_MODE_VALUE_EARLIEST = "earliest-offset";
private static final String SCAN_STARTUP_MODE_VALUE_LATEST = "latest-offset";
private static final String SCAN_STARTUP_MODE_VALUE_SPECIFIC_OFFSET = "specific-offset";
private static final String SCAN_STARTUP_MODE_VALUE_TIMESTAMP = "timestamp";
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use enum replace switch-case.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reference mysql pipeline,it like this

Comment on lines +185 to +196
builder.hostname(config.getOptional(OracleDataSourceOptions.HOSTNAME).get())
.port(config.getOptional(OracleDataSourceOptions.PORT).get())
.database(
config.getOptional(OracleDataSourceOptions.DATABASE)
.get()) // monitor database
.schemaList(
config.getOptional(OracleDataSourceOptions.SCHEMALIST)
.get()) // monitor schema
.tableList(capturedTables) // monitor
// EMP table
.username(config.getOptional(OracleDataSourceOptions.USERNAME).get())
.password(config.getOptional(OracleDataSourceOptions.PASSWORD).get())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use config.get(ConfigOption option)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +213 to +243
<shadedPattern>
com.ververica.cdc.connectors.shaded.org.apache.kafka
</shadedPattern>
</relocation>
<relocation>
<pattern>org.antlr</pattern>
<shadedPattern>
com.ververica.cdc.connectors.shaded.org.antlr
</shadedPattern>
</relocation>
<relocation>
<pattern>com.fasterxml</pattern>
<shadedPattern>
com.ververica.cdc.connectors.shaded.com.fasterxml
</shadedPattern>
</relocation>
<relocation>
<pattern>com.google</pattern>
<shadedPattern>
com.ververica.cdc.connectors.shaded.com.google
</shadedPattern>
</relocation>
<relocation>
<pattern>com.esri.geometry</pattern>
<shadedPattern>com.ververica.cdc.connectors.shaded.com.esri.geometry</shadedPattern>
</relocation>
<relocation>
<pattern>com.zaxxer</pattern>
<shadedPattern>
com.ververica.cdc.connectors.shaded.com.zaxxer
</shadedPattern>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why relocate com.ververica.cdc.connectors?
Please use the correct package path.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +18 to +19
package com.apache.flink.cdc.connectors.oracle.dto;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use org.apache.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

Comment on lines +32 to +34
import com.apache.flink.cdc.connectors.oracle.source.OracleDataSource;
import com.apache.flink.cdc.connectors.oracle.source.OracleDataSourceOptions;
import com.apache.flink.cdc.connectors.oracle.utils.OracleSchemaUtils;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use org.apache.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

import java.util.Set;
import java.util.stream.Collectors;

import static com.apache.flink.cdc.connectors.oracle.source.OracleDataSourceOptions.METADATA_LIST;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use org.apache.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have modified

}
throw new IllegalArgumentException(
String.format(
"[%s] cannot be found in mysql metadata.",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oracle metadata

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for review!have modified!

@linjianchang linjianchang force-pushed the master-36796 branch 2 times, most recently from 0c77212 to 0237401 Compare August 5, 2025 10:39
Comment on lines +109 to +112
dbzProperties.setProperty(
"snapshot.locking.mode", config.get(OracleDataSourceOptions.SNAPSHOT_LOCKING_MODE));
dbzProperties.setProperty(
"snapshot.locking.mode", config.get(OracleDataSourceOptions.SNAPSHOT_LOCKING_MODE));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a duplicate code snippet.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have remove the duplicate code

Comment on lines +92 to +94
@Override
public EventSourceProvider getEventSourceProvider() {
String url = config.get(OracleDataSourceOptions.JDBC_URL);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should null parameters be handled?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

url could be null,and hostname,port,database,username,password must be not null,have add checknotnull

Comment on lines +208 to +214
switch (modeString.toLowerCase()) {
case SCAN_STARTUP_MODE_VALUE_INITIAL:
return StartupOptions.initial();

case SCAN_STARTUP_MODE_VALUE_LATEST:
return StartupOptions.latest();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should other situations of StartupOptions be handled?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have added the remaining supported StartupOptions

Comment on lines +87 to +97
public static final ConfigOption<String> SERVER_TIME_ZONE =
ConfigOptions.key("server-time-zone")
.stringType()
.noDefaultValue()
.withDescription(
"The session time zone in database server. If not set, then "
+ "ZoneId.systemDefault() is used to determine the server time zone.");

public static final ConfigOption<String> SERVER_ID =
ConfigOptions.key("server-id")
.stringType()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does Oracle CDC have these configurations?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have remove SERVER_ID,and SERVER_TIME_ZONE is need

Comment on lines +144 to +148
@Override
protected Object convertToString(Object dbzObj, Schema schema) {
// the Geometry datatype in oracle will be converted to
// a String with Json format
if (Point.LOGICAL_NAME.equals(schema.name())
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if Oracle also needs to be handled in this way.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oracle database provides powerful geometric data type support through the Spatial module, which is mainly based on the SDO_GEOMETRY object type.

@aiwenmo
Copy link
Copy Markdown
Contributor

aiwenmo commented Oct 12, 2025

Hi. Thanks for your contribution. @linjianchang

I've found that some code might be useless in Oracle CDC. Please simplify the code carefully to make it as concise as possible.

@Mrart
Copy link
Copy Markdown
Contributor

Mrart commented Dec 15, 2025

and we need implement e2e OracleE2eITCase like PostgresE2eITCase.

Aleady add OracleE2eITCase

LGTM. cc @LYanquan

@lvyanquan lvyanquan self-assigned this Dec 15, 2025
@lvyanquan lvyanquan added this to the V3.6.0 milestone Dec 25, 2025
Copy link
Copy Markdown
Contributor

@lvyanquan lvyanquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @linjianchang's contribution, left some comments.

@@ -0,0 +1,15 @@
# Copyright 2023 Ververica Inc.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use ASF License.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use ASF License.

modified


# Set root logger level to OFF to not flood build logs
# set manually to INFO for debugging purposes
rootLogger.level=INFO
Copy link
Copy Markdown
Contributor

@lvyanquan lvyanquan Dec 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to set this to ERROR to reduce log in CI machine.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to set this to ERROR to reduce log in CI machine.

modified

@@ -0,0 +1,266 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2023 Ververica Inc.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use ASF license.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use ASF license.

modified

@@ -0,0 +1,26 @@
################################################################################
# Copyright 2023 Ververica Inc.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Maybe you could modify all files at once.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto. Maybe you could modify all files at once.

modified

.durationType()
.defaultValue(Duration.ofSeconds(30))
.withDescription(
"Optional interval of sending heartbeat event for tracing the latest available binlog offsets");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

binlog?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

binlog?

modified

ConfigOptions.key("debezium.database.connection.adapter")
.stringType()
.defaultValue("logminer")
.withDescription("Database connection adapter.");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should introduce all available values.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should introduce all available values.

modified


/** IT tests for {@link OracleDataSource}. */
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OraclePipelineITCase extends OracleSourceTestBase {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's necessary to add some tests to cover the scene of restoring from savepoint with different start up mode.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's necessary to add some tests to cover the scene of restoring from savepoint with different start up mode.

added

String str = (String) dbzObj;
// TIMESTAMP_LTZ type is encoded in string type
Instant instant = Instant.parse(str);
Instant instant = ZonedTimestamp.FORMATTER.parse(str, Instant::from);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some explanation about why this change is necessary?

Copy link
Copy Markdown
Contributor Author

@linjianchang linjianchang Jan 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add some explanation about why this change is necessary?

The parsing of the time zone type for Oracle failed
企业微信截图_17679489006213
All sources in the pipeline have been tested with fulltype's ut and are normal. This change has not affected other connectors.
企业微信截图_17679538297312
企业微信截图_17679536267111

+ " port: %d\n"
+ " username: %s\n"
+ " password: %s\n"
+ " tables: %s.PRODUCTS\n"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only one table here, you can add a few more tables.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is only one table here, you can add a few more tables.

modified

return DataTypes.STRING();
}
if (scale < 0 || scale > 36) {
return DataTypes.STRING();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain why this change is necessary.

Copy link
Copy Markdown
Contributor Author

@linjianchang linjianchang Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please explain why this change is necessary.

企业微信截图_17679497824493 "Does Oracle's 'nember' type have a Scala value?" It is in the range of 0-36 in decimal, so it is not in the decimal range Convert to string

All sources in the pipeline have been tested with fulltype's ut and are normal. This change has not affected other connectors.
企业微信截图_17679536267111
企业微信截图_17682126247568

Comment on lines +74 to +84
OracleDialect oracleDialect = new OracleDialect();
OracleEventDeserializer deserializer =
new OracleEventDeserializer(
DebeziumChangelogMode.ALL,
config.get(OracleDataSourceOptions.SCHEMA_CHANGE_ENABLED),
readableMetadataList);

RedoLogOffsetFactory offsetFactory = new RedoLogOffsetFactory();
OracleTableSourceReader oracleChangeEventSource =
new OracleTableSourceReader(
configFactory, deserializer, offsetFactory, oracleDialect);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be updated to the following code to avoid Raw use of parameterized class.

        OracleEventDeserializer<Event> deserializer =
                new OracleEventDeserializer<>(
                        DebeziumChangelogMode.ALL,
                        config.get(OracleDataSourceOptions.SCHEMA_CHANGE_ENABLED),
                        readableMetadataList);

        RedoLogOffsetFactory offsetFactory = new RedoLogOffsetFactory();
        OracleTableSourceReader<Event> oracleChangeEventSource =
                new OracleTableSourceReader<>(
                        configFactory, deserializer, offsetFactory, oracleDialect);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can be updated to the following code to avoid Raw use of parameterized class.

        OracleEventDeserializer<Event> deserializer =
                new OracleEventDeserializer<>(
                        DebeziumChangelogMode.ALL,
                        config.get(OracleDataSourceOptions.SCHEMA_CHANGE_ENABLED),
                        readableMetadataList);

        RedoLogOffsetFactory offsetFactory = new RedoLogOffsetFactory();
        OracleTableSourceReader<Event> oracleChangeEventSource =
                new OracleTableSourceReader<>(
                        configFactory, deserializer, offsetFactory, oracleDialect);

Modified

import static org.apache.flink.cdc.connectors.base.utils.SourceRecordUtils.isSchemaChangeEvent;

/** The {@link RecordEmitter} implementation for Oracle pipeline connector. */
public class OraclePipelineRecordEmitter<T> extends IncrementalSourceRecordEmitter<T> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can clarify that the generic type T here is Event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can clarify that the generic type T here is Event.

Modified

* and Watermark Signal Algorithm which supports parallel reading snapshot of table and then
* continue to capture data change by streaming reading.
*/
public class OracleTableSourceReader<T> extends OracleSourceBuilder.OracleIncrementalSource<T> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can clarify that the generic type T here is Event.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can clarify that the generic type T here is Event.

Modified

if (isDataChangeRecord || isSchemaChangeEvent(element)) {
TableId debeziumTableId = getTableId(element);
TableId tableId =
TableId.parse(debeziumTableId.schema() + "." + debeziumTableId.table());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use debeziumTableId directly here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use debeziumTableId directly here?

Modified

sendCreateTableEvent(jdbc, tableId, output);
}
alreadySendCreateTableTables.add(
TableId.parse(tableId.schema() + "." + tableId.table()));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alreadySendCreateTableTables.add(new TableId(tableId.schema(), null, tableId.table()));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

alreadySendCreateTableTables.add(new TableId(tableId.schema(), null, tableId.table()));

Modified


private static final Logger LOGGER =
LoggerFactory.getLogger(
io.debezium.connector.oracle.antlr.listener.AlterTableParserListener.class);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoggerFactory.getLogger(OracleAlterTableParserListener.class);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoggerFactory.getLogger(OracleAlterTableParserListener.class);

Modified

.booleanType()
.defaultValue(true)
.withDescription(
"Whether send schema change events, by default is true. If set to false, the schema changes will not be sent.");
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen if we alter table structure but setting schema-change.enabled to false? Should we always set this parameter to true?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen if we alter table structure but setting schema-change.enabled to false? Should we always set this parameter to true?
always set this parameter to true

/**
* Test utility for creating converter, formatter and deserializer of a table in the test database.
*/
public class TestTable {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

already removed

import static org.apache.flink.util.Preconditions.checkState;

/** Oracle test utilities. */
public class OracleTestUtils {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

already removed

import java.util.stream.Collectors;

/** Formatter that formats the {@link SourceRecord} to String. */
public class RecordsFormatter {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class seems to be useless.

already removed

throw new IllegalArgumentException(
"Cannot find any table by the option 'tables' = " + tables);
}
configFactory.tableList(capturedTables);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this configuration mode, we cannot handle tables newly created during the incremental phase. Perhaps you can refer to the implementation of MySQL.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this configuration mode, we cannot handle tables newly created during the incremental phase. Perhaps you can refer to the implementation of MySQL.

Remove it temporarily. We won't add this feature in this version

@linjianchang linjianchang force-pushed the master-36796 branch 3 times, most recently from 08a8f61 to a6cabd0 Compare January 13, 2026 07:50
@github-actions github-actions Bot removed the base label Jan 23, 2026

@Override
public void enterModify_col_properties(PlSqlParser.Modify_col_propertiesContext ctx) {
resolveColumnDataType(ctx);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If dataType is not specified, will changing the column attribute to nullable have no effect?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this case, could we add a drop column event? Drop column seems simply makes the column nullable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If dataType is not specified, will changing the column attribute to nullable have no effect?

Why is there a situation where no data type is specified? If no new data type is specified in the MODIFY clause, Oracle will report an error

Copy link
Copy Markdown
Contributor

@zml1206 zml1206 Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ALTER TABLE t MODIFY a NULL;
Is this syntax not supported?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lvyanquan
Copy link
Copy Markdown
Contributor

Please update https://github.com/apache/flink-cdc/blob/master/.github/workflows/modules.py to include test for this newly added module.

@linjianchang
Copy link
Copy Markdown
Contributor Author

Please update https://github.com/apache/flink-cdc/blob/master/.github/workflows/modules.py to include test for this newly added module.

Have updated the file in master-36796

Copy link
Copy Markdown
Contributor

@lvyanquan lvyanquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@lvyanquan lvyanquan merged commit c8e444b into apache:master Feb 10, 2026
28 of 29 checks passed
public static List<TableId> listTables(
OracleSourceConfig sourceConfig, @Nullable String dbName) {
try (JdbcConnection jdbc = createOracleConnection(sourceConfig)) {
List<String> databases =
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CDB + PDB mode can not find PDB databases here, should change session to PDB through setSessionToPdb(${pdbName}) before listDatabase?

ThorneANN pushed a commit to ThorneANN/flink-cdc that referenced this pull request Mar 19, 2026
Mrart pushed a commit to Mrart/flink-cdc that referenced this pull request Mar 26, 2026
Set<TableId> tableIdSet = new HashSet<>();
String queryTablesSql =
"SELECT OWNER ,TABLE_NAME,TABLESPACE_NAME FROM ALL_TABLES \n"
+ "WHERE TABLESPACE_NAME IS NOT NULL AND TABLESPACE_NAME NOT IN ('SYSTEM','SYSAUX') "
Copy link
Copy Markdown

@sumitnekar sumitnekar Apr 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oracle returns table space as null for partitioned tables. When I tested with a partitioned table, this table was never discovered and initial snapshot never started. So it was throwing generic error - “Db history topic or its content is fully or partially missing. Please check ur database history topic configuration”.

But it worked fine for me with
SourceFunction-based DataStream API. It never worked for me with Incremental Snapshot based DataStream API.

@linjianchang @lvyanquan Let me know If I am missing something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants