Enable Polaris to "catalog" tables based on Apache Flink CDC tables

[georgelza]

Is your feature request related to a problem? Please describe.

No response

Describe the solution you'd like

We need a way to allow the user to create a catalog, type: generic
with catalog type being polaris.
Which then point at a PostgreSQL source table to consume, or mySQL, or Mongo or Kafka or ...
This way if the user closes the session and come back in his CDC sources are all still defined inside a catalog and one or more databases,

Describe alternatives you've considered

Currently only Hive Metastore allows for storing CDC source tables.

Additional context

No response

[rishii-19-works]

Hi @georgelza thanks for raising this — the use case makes sense, especially for CDC sources where table definitions shouldn’t disappear after a session ends.

I’m interested in contributing here and wanted to confirm my understanding. From what I see, the goal is to persist CDC source table metadata (e.g. from Flink CDC) in a Polaris catalog so they can be discovered and managed like other catalog tables.

Before starting, could you clarify whether CDC tables should be treated as first-class catalog entities or as external references with stored metadata? Also are there any existing extension points in Polaris for integrating non-Iceberg sources?

Thanks — happy to explore the codebase once I have some direction.

[georgelza]

hi hi,
glad you see / understand the use case.
as for the other points, I can't comment... Those facts will need to come from the Polaris project team.
I'd like to think external tables as used for source data using CDC framework to be first class, but have to be honest and probably think this would be more metadata only, defining how where they exist, so that a user can come back in, do a use catalog/database and show tables and there they are, without having to be redefined.
... the potential list of sources here can be extensive...
PostgreSQL,
MySQL,
Kafka
Mongo
Iceberg
Paimon
REDIS
... and I'd imagine any other sources CDC support