Skip to content

Conversation

nickdelnano
Copy link

This PR adds a new format for defining table properties as specific.table.properties.<database_name>.<table_name>.<property>. This allows defining properties that are specific to a table. The table specific property takes precedence.

Paimon sink supports defining table properties as table.properties.*. They apply to ALL tables. In some cases it is warranted to set table properties based on the table use case. Some examples are compaction settings, Iceberg compatibility settings, etc. This PR allows this.

@nickdelnano nickdelnano changed the title Paimon connector - support table specific table.properties config [pipeline-connector/paimon] Support table specific table.properties configuration Aug 6, 2025
@nickdelnano
Copy link
Author

@lvyanquan can you review this change?

@lvyanquan
Copy link
Contributor

We can modify the table options through transform, but I agree to provide more configurations in the sink as alternative solutions, refer to https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/core-concept/transform/#specify-table-creation-configuration.

Compared to transform, the current configuration requires too many parameters to be set for situations where many tables need to be parameterized(users usually synchronize a large number of tables in one job). Can we provide a solution for parameter configuration based on regular expressions (Selectors)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants