Connection pooling in ducklake

[sjoe77]

Hello, I am not sure what is a proper way to create a connection pool for duckdb with ducklake extension. I notice that after running for some time properly, I am seeing these kind of errors which indicative of bad connection in the pool. Restarting the server helps but the problem come back as connection are used and released. its entirely possible my connection pool implementation has a bug.

detail": "UNKNOWN: [FlightSQL] Catalog Error: Failed to get data file list from DuckLake: Table with name ducklake_data_file does not exist!\nDid you mean \"__ducklake_metadata_my_ducklake.ducklake_data_file\"?\n\nLINE 3: FROM \"__ducklake_metadata_auth_data\".\"public\".ducklake_data_file...\n .

Thanks in advance

[staticlibs]

@sjoe77

Hi, I am assuming you are implementing connection pool in a C++ extension? I wonder whether it is necessary to use custom extension in your setup, would it be possible to use some existing generic connection pool lib on "client" (Python, Java etc) side instead?

[sjoe77]

db_pool.txt
I am doing it in python. See attached db_pool.py

[staticlibs]

@sjoe77

Table with name ducklake_data_file does not exist

I did not run it (config.py is required), just this error looks suspiciously like the wrong current catalog. And there is USE memory in pool.release() call. Perhaps the pool is working, just the ducklake_data_file DuckLake table needs to be qualified with DuckLake alias (or USE <ducklake_alias> to be added in acquire)?

[sjoe77]

when I init the connections in the pool I attach a couple of databases on each connection. The current database will always be MEMORY. When I query using a connection from the pool, I always using fully qualified name for tables or views. For example Select * from dbname.schemaname.tablename.

[staticlibs]

@sjoe77

This invalidates my guesses. Still the error looks a bit weird:

__ducklake_metadata_my_ducklake.ducklake_data_file
__ducklake_metadata_auth_data"."public".ducklake_data_file

Some kind of mix up between multiple DuckLake instances is going on. Another guess: assuming from the pool config that the catalog is in Postgres, perhaps you are hitting #283?

[sjoe77]

Yes. It's possible that it's related to the postgres connector.

…

On Thu, Jul 17, 2025, 5:38 PM Alex Kasko ***@***.***> wrote: @sjoe77 <https://github.com/sjoe77> This invalidates my guesses. Still the error looks a bit weird: __ducklake_metadata_my_ducklake.ducklake_data_file __ducklake_metadata_auth_data"."public".ducklake_data_file Some kind of mix up between multiple DuckLake instances is going on. Another guess: assuming from the pool config that the catalog is in Postgres, perhaps you are hitting #283 <#283>? — Reply to this email directly, view it on GitHub <#299 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALFTAJDWS6OO6HWVYXW6PI33JAJVHAVCNFSM6AAAAACBX67TQGVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGNZZHAYTQOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[sjoe77]

A single connection initialized on thread local pattern works as a workaround but there is a performance penalty

[sjoe77]

Happy to report my issue is fixed. I updated duckdb from 1.3.1 to 1.3.2 which fixed it. https://github.com/duckdb/duckdb/releases/tag/v1.3.2,

Key Connection/Concurrency Fixes:

• SharedLockTable in DataTable::Fetch: Better concurrent access handling
• Lock management: "Grab lock before finalizing dynamic filters" prevents race conditions
• Bulk queue operations: "Implement bulk enqueue for non-concurrent queue" improves thread safety
• Exception handling: "Print internal exception stack traces on failed transaction rollback" and internal exception handling in compression paths

My guess is most Relevant Fix:
The SharedLockTable improvement is probably the biggest factor - it changes how DuckDB handles concurrent data access, which would directly impact your connection pooling
scenario where multiple connections might be accessing the same data simultaneously.