Skip to content

[oracle] Support specific-offset startup mode with SCN for Oracle pipeline source - #4510

Open
fightBoxing wants to merge 1 commit into
apache:masterfrom
fightBoxing:oracle-specific-offset-scn
Open

[oracle] Support specific-offset startup mode with SCN for Oracle pipeline source#4510
fightBoxing wants to merge 1 commit into
apache:masterfrom
fightBoxing:oracle-specific-offset-scn

Conversation

@fightBoxing

Copy link
Copy Markdown

Motivation

The Oracle pipeline connector currently only supports initial / snapshot / latest-offset startup modes, while the MySQL pipeline connector already supports specific-offset. For Oracle users who want to skip the full snapshot phase and start incremental (LogMiner redo log) consumption from a known position, there is no way to specify a starting SCN.

Note: the underlying Debezium property log.mining.start.position.scn cannot be used as a workaround, because the shaded Debezium dependency in the connector jar does not include that configuration, and passing it through the debezium.* prefix is silently ignored.

Modification

  • OracleDataSourceOptions: add new option scan.startup.specific-offset.scn (Long, no default value)
  • OracleDataSourceFactory: add specific-offset branch in getStartupOptions(), which builds StartupOptions.specificOffset(Map{scn, commit_scn}). The base framework (StreamSplitAssigner#createStreamSplit) already natively consumes SPECIFIC_OFFSETS offsets, so no framework change is needed.

Usage

source:
  type: oracle
  scan.startup.mode: specific-offset
  scan.startup.specific-offset.scn: 2053676

Verification

Verified end-to-end on a real cluster (Flink 1.20.1 YARN session + Oracle 12c LogMiner + Iceberg sink):

  • JobManager log confirms the stream split starts exactly at the specified SCN: Assign split StreamSplit{offset={commit_scn=0, scn=2053676}}
  • Only changes committed after the specified SCN are captured; all pre-existing snapshot rows are correctly skipped

…eline source

The Oracle pipeline connector only supports initial/snapshot/latest-offset startup modes, while the MySQL connector already supports specific-offset. This adds specific-offset mode to the Oracle DataSource, allowing users to start incremental reading (LogMiner redo log consumption) from a specified SCN via scan.startup.specific-offset.scn.

Changes: add SCAN_STARTUP_SPECIFIC_OFFSET_SCN option in OracleDataSourceOptions; add specific-offset branch in getStartupOptions() of OracleDataSourceFactory, building StartupOptions.specificOffset with {scn, commit_scn} map which is natively consumed by the base framework StreamSplitAssigner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant