Replies: 1 comment
-
I actually solved this using pgbouncer to attach datalake each time, not sure if there any better solutions. Also looking to see best way to have the clients of postgres to see the datalake tables naturally in public schema. Right now I just wrapped each table with a view selecting r[''] so that the view has the actual column definitions. Hoping for a better way? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
We're using pg_duckdb with DuckLake to query Parquet files stored in Cloudflare R2 (S3-compatible). We've successfully implemented a hybrid warehouse where:
Current Implementation
Users must call attach_ducklake() once per session before querying:
CREATE OR REPLACE FUNCTION attach_ducklake()
RETURNS text
LANGUAGE plpgsql
AS $$
DECLARE
catalog_path text := '/var/lib/postgresql/data/ducklake_catalog.duckdb';
bucket_name text := 'duckdb';
BEGIN
-- Load extensions
PERFORM duckdb.install_extension('httpfs');
PERFORM duckdb.load_extension('httpfs');
PERFORM duckdb.install_extension('aws');
PERFORM duckdb.load_extension('aws');
PERFORM duckdb.install_extension('ducklake');
PERFORM duckdb.load_extension('ducklake');
END;
$$;
Typical user workflow:
-- Every new connection requires this
SELECT attach_ducklake();
-- Then queries work via views
SELECT COUNT(*) FROM "Order"; -- View wraps duckdb.query('SELECT * FROM my_lake."Order"')
Problem
ATTACH is session-scoped - it doesn't persist across connections. This creates friction for:
Question
Is there a way to make the DuckLake catalog attachment persistent across all connections?
Options we're considering:
Ideal Behavior
We'd love for the catalog to be automatically available to all connections without requiring manual attach_ducklake() calls, similar to how foreign data wrappers or schemas work in PostgreSQL.
Environment
Any guidance would be greatly appreciated! We're happy to contribute documentation or examples if we find a good solution.
Beta Was this translation helpful? Give feedback.
All reactions