Skip to content

Support single-catalog (standard DuckLake) on Postgres #165

Description

@shefeek-jinnah

Summary

Add a single-catalog Postgres metadata writer that produces standard, spec-compliant DuckLake catalogs (unscoped, relative paths, no multicatalog tables), complementing the existing multicatalog Postgres writer.

Background

Today Postgres is multicatalog-only:

  • PostgresMetadataWriter is bound to a catalog_id at construction and always scopes schema paths (schema.path = cat_<id>/<schema>, metadata_writer_postgres.rs), and it creates the custom multicatalog tables (ducklake_catalog, ducklake_catalog_schema_map, ducklake_catalog_snapshot_map) plus catalog_id columns.
  • The other backends (SQLite, MySQL, DuckDB) are single-catalog and write the standard DuckLake layout — unscoped, relative.
  • A single-catalog Postgres reader already exists (PostgresMetadataProvider), but there's no matching single-catalog Postgres writer.

Motivation

  • Spec compliance / relocatability. DuckLake is relative-by-default; a single-catalog Postgres writer would produce fully relative, relocatable catalogs, matching the spec exactly.
  • Interoperability. Standard DuckLake-on-Postgres would be interoperable with other tools that speak the spec (e.g. DuckDB's ducklake extension using a Postgres catalog).
  • Symmetry. Brings Postgres in line with the other single-catalog backends.

Proposed approach

A separate single-catalog Postgres writer (rather than making PostgresMetadataWriter dual-mode), modeled on MySqlMetadataWriter — the SQL is nearly identical:

  • writes the standard DuckLake metadata schema (no catalog_id columns, no map tables),
  • catalog_id() returns None, so the table writer's existing branch (table_writer.rsNone => base_key_path) places files unscoped, and
  • schema/table/file paths stored unscoped and relative.

It pairs with the existing PostgresMetadataProvider reader. Most of the plumbing (file placement, reader) already exists; the main work is the writer's standard-layout DDL and inserts.

Scope / non-goals

This issue covers only the single-catalog Postgres writer. Promoting/adopting an existing single-catalog catalog into multicatalog (which would create a mixed old-unscoped / new-scoped file layout) is a separate follow-up.

Relation to #158

#158 handles reading a schema whose stored path is unscoped while new files are written scoped — a mixed layout. In the current code that state isn't reachable, because the multicatalog writer always scopes. It would become reachable only once we (a) support single-catalog Postgres (this issue) and (b) add a path to adopt/promote such a catalog into multicatalog while keeping its existing files in place. Tracking that dependency here so the two efforts stay aligned.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions