[oracle] Support specific-offset startup mode with SCN for Oracle pipeline source - #4510
Open
fightBoxing wants to merge 1 commit into
Open
[oracle] Support specific-offset startup mode with SCN for Oracle pipeline source#4510fightBoxing wants to merge 1 commit into
fightBoxing wants to merge 1 commit into
Conversation
…eline source
The Oracle pipeline connector only supports initial/snapshot/latest-offset startup modes, while the MySQL connector already supports specific-offset. This adds specific-offset mode to the Oracle DataSource, allowing users to start incremental reading (LogMiner redo log consumption) from a specified SCN via scan.startup.specific-offset.scn.
Changes: add SCAN_STARTUP_SPECIFIC_OFFSET_SCN option in OracleDataSourceOptions; add specific-offset branch in getStartupOptions() of OracleDataSourceFactory, building StartupOptions.specificOffset with {scn, commit_scn} map which is natively consumed by the base framework StreamSplitAssigner.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The Oracle pipeline connector currently only supports
initial/snapshot/latest-offsetstartup modes, while the MySQL pipeline connector already supportsspecific-offset. For Oracle users who want to skip the full snapshot phase and start incremental (LogMiner redo log) consumption from a known position, there is no way to specify a starting SCN.Note: the underlying Debezium property
log.mining.start.position.scncannot be used as a workaround, because the shaded Debezium dependency in the connector jar does not include that configuration, and passing it through thedebezium.*prefix is silently ignored.Modification
OracleDataSourceOptions: add new optionscan.startup.specific-offset.scn(Long, no default value)OracleDataSourceFactory: addspecific-offsetbranch ingetStartupOptions(), which buildsStartupOptions.specificOffset(Map{scn, commit_scn}). The base framework (StreamSplitAssigner#createStreamSplit) already natively consumes SPECIFIC_OFFSETS offsets, so no framework change is needed.Usage
Verification
Verified end-to-end on a real cluster (Flink 1.20.1 YARN session + Oracle 12c LogMiner + Iceberg sink):
Assign split StreamSplit{offset={commit_scn=0, scn=2053676}}