Skip to content

Conversation

xxntti3n
Copy link

@xxntti3n xxntti3n commented Sep 9, 2025

This PR adds support for per-table chunk key column configuration in the PostgreSQL CDC connector, enabling fine-grained control over incremental snapshot
chunking. Previously, all tables shared the same chunk key column, which was inefficient for heterogeneous table schemas.

Key Changes

  • ✨ Added chunk key column parsing logic to PostgreSQL DataSourceFactory
  • 🔧 Implemented comma-separated configuration format support

Example Usage

sourceConf:
scan.incremental.snapshot.chunk.key-column: public.action_logs:created_at,public.service_logs:created_at

@lvyanquan
Copy link
Contributor

Please add more description about this change.

@Mrart
Copy link
Contributor

Mrart commented Sep 19, 2025

@xxntti3n I'm trying to understand what you want to do, do you want to configure a separate chunk key column for schema.table? What's wrong with the previous chunkkeycolumn configuration?

@xxntti3n xxntti3n force-pushed the universal-chunk-column-dev branch 4 times, most recently from 5314014 to 4fc6319 Compare September 23, 2025 16:45
@xxntti3n
Copy link
Author

xxntti3n commented Sep 25, 2025

@xxntti3n I'm trying to understand what you want to do, do you want to configure a separate chunk key column for schema.table? What's wrong with the previous chunkkeycolumn configuration?

yes i want to conf with multiple tables like mysql dialect . Example Conf : scan.incremental.snapshot.chunk.key-column: public.table1:created_at,public.table2:created_at . Currently, Postgres only support 1 table

@xxntti3n xxntti3n force-pushed the universal-chunk-column-dev branch from 2742ce7 to 956c4e0 Compare September 25, 2025 07:44
@github-actions github-actions bot added the docs Improvements or additions to documentation label Sep 25, 2025
@xxntti3n
Copy link
Author

Hi @lvyanquan, @Mrart - I've added the PR description and would appreciate your review. Looking forward to your feedback. Thanks!

@xxntti3n xxntti3n changed the title add handle chunkKeyColumn for Postgres Implement comma-separated parsing for chunk key columns Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

base docs Improvements or additions to documentation postgres-cdc-connector

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants