REST catalogue issues

[boudica-dev-eng]

Non-standard REST Catalogue paths

The hardcoded path strings for the REST Catalogue implementation are not standardised to the Iceberg REST Catalogue specification. pg_lake adds /api/catalog/ to the front of every path before following the standard spec, preventing most REST Catalogue implementations such as Lakekeeper from functioning.

See https://github.com/Snowflake-Labs/pg_lake/blob/main/pg_lake_iceberg/include/pg_lake/rest_catalog/rest_catalog.h#L37

You can check here on the official spec that every path start from /v1/ onwards, allow maximum compatibility: https://editor.swagger.io/?url=https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml

Unauthenticated connections

Furthermore, the current implementation prevents unauthenticated requests for testing. When trying to perform an unauthenticated connection to an existing Iceberg catalogue in Lakekeeper, the server demands that pg_lake_iceberg.rest_catalog_client_id must be set.

Documentation

Lastly, it's very unclear how to configure everything. Which configs apply to pgduck_server, and which to PostgreSQL? Deploying to CNPG has been very difficult and I have had to proxy the Unix Socket to an IP connection to allow for deployment on Kubernetes.

I'm still unsure how to specify which warehouse in Lakekeeper that I would like pg_lake to connect.

Great project. Happy to help any way I can if testing is needed.

[sfc-gh-mslot]

Thanks for the feedback! The reason it's not yet documented is because these things are still in flux as we work on improving the configuration mechanism, support for more catalogs, and better handling of catalog failures.