Skip to content

Reuse DuckDB connection for metadata queries instead of creating new connection per call#43

Merged
shefeek-jinnah merged 2 commits into
mainfrom
shefeek/duckdb_connection_caching
Jan 16, 2026
Merged

Reuse DuckDB connection for metadata queries instead of creating new connection per call#43
shefeek-jinnah merged 2 commits into
mainfrom
shefeek/duckdb_connection_caching

Conversation

@shefeek-jinnah
Copy link
Copy Markdown
Collaborator

@shefeek-jinnah shefeek-jinnah commented Jan 16, 2026

Summary

  • Refactored DuckdbMetadataProvider to use a single shared DuckDB connection protected by Arc<Mutex<Connection>> instead of creating a new connection for each metadata query
  • Reduces connection creation overhead for read-only metadata operations

#44

Changes

  • Changed struct field from catalog_path: String to conn: Arc<Mutex<Connection>>
  • Connection is now created once in new() and reused across all metadata calls
  • Added connection() method that returns MutexGuard<Connection> for thread-safe access
  • Renamed open_connection_with_path() to create_connection() for clarity

Rationale

Previously, each of the 14+ metadata methods (get_current_snapshot, list_schemas, get_table_by_name, etc.) opened a new DuckDB connection. Since all operations are read-only and metadata queries are fast, a mutex-protected shared connection is sufficient and eliminates unnecessary connection overhead.

@shefeek-jinnah shefeek-jinnah marked this pull request as ready for review January 16, 2026 09:00
@shefeek-jinnah shefeek-jinnah merged commit c103573 into main Jan 16, 2026
4 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant