fix: make DuckDB attachments logic more robust

[sgrebnov]

PR fixes DuckDB errors caused by a race condition when multiple connections attempt to attach databases concurrently.

When multiple connections called query_arrow() simultaneously, each would:

1. Check if attachments exist via PRAGMA database_list
2. If not found, create new DuckDBAttachments with a unique random ID
3. Run ATTACH IF NOT EXISTS '{db}' AS attachment_{random_id}_{i} one by one

Race condition: Between step 1 (check) and steps 3 (attach one by one), another connection could attach the same file or retrieve only partially attached databases. This result into errors

Spice query failed. Status: 400, body: Execution error: Failed to execute query.\nDuckDB connection failed.\nBinder Error: Unique file handle conflict: Cannot attach "attachment_SDFrVyz5_0" - the database file "/app/.spice/data/accelerated_duckdb.db" is already attached by database "attachment_23yXahuv_0"\nFor details, refer to the DuckDB manual: https://duckdb.org/docs/"}

2025-11-17T19:30:47.876652Z WARN datafusion_table_providers::sql::db_connection_pool::dbconnection::duckdbconn: my_table.duckdb not found among existing attachments

All DuckDB connections acquired from a single pool (including its clones) or created via try_clone() share the same catalog including attached databases, but not the search_path which is connection-level setting.

┌─────────────────────────────────────────────────────────┐
│                  duckdb_database (self.db)              │
│  ┌─────────────────────────────────────────────────┐    │
│  │              Catalog (shared state)              │    │
│  │  - Tables                                        │    │
│  │  - Attached databases (attachment_xxx_0, etc.)   │    │
│  │  - Search paths (per-connection setting)         │    │
│  └─────────────────────────────────────────────────┘    │
│                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │ Connection 1 │  │ Connection 2 │  │ Connection 3 │   │
│  │  (original)  │  │ (try_clone)  │  │ (pool.get())  │   │
│  │              │  │              │  │              │   │
│  │ search_path: │  │ search_path: │  │ search_path: │   │
│  │   "main"     │  │   (default)  │  │   (default)  │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
└─────────────────────────────────────────────────────────┘

Solution

1. Share DuckDBAttachments across pool clones using Arc<OnceCell<...>> to ensure database attachments are configured exactly once per underlying connection pool (with first set_database_attachments wins, all datasets using the same pool has the same attachments)

• this guarantees that only single instance ofDuckDBAttachments exist per pool and its clones.
• we can't pass attachments as parameter for pool initialization as attachments are not always available/calculated when pool is created first time.

2. Similar to pool use Arc<OnceCell for search_path to guarantee that actual logic to apply attachments is executed once. This also leads to better performance as we execute attach only once and re-use cached search path w/o executing additional statements to retrieve/verify/parse existing configuration before each call.

Other alternatives considered:

• Apply attachments immediately when set_attachments is called - unfortunately, at that moment not all tables are created and they won't be added to the catalog. Approach above do attach as part of first query, with combination of ready state this could be considered as very robust approach (all tables exist).
• Apply attachments as part of connection setup, not query logic - pool proactively creates connections so this leads to the same issue as above
• There still could be cases where attach happens when not all tables exist, for example - one of accelerated tables with different duckdb_file can't connect to source during initialization to get schema and create initial empty table and first table is fully accelerated and queried - in this case first table will initialize attachments, but actual federated query will fail later (same for view that can start initialization later). This is not related to this specific implementation/change, just a note, that this could be further improved, for example by using additional step to update metadata when all tables are loaded. Current approach has been selected as robust, simple to implement and covering all important cases.