Skip to content

Conversation

@sgrebnov
Copy link
Collaborator

@sgrebnov sgrebnov commented Dec 4, 2025

Note: draft is I found out that DuckDB does not recognize created tables after file was attached so this should be done when all accelerated tables are available.

Fixes DuckDB errors caused by a race condition when multiple connections attempt to attach databases concurrently.

When multiple connections called query_arrow() simultaneously, each would:

  1. Check if attachments exist via PRAGMA database_list
  2. If not found, create new DuckDBAttachments with a unique random ID
  3. Run ATTACH IF NOT EXISTS '{db}' AS attachment_{random_id}_{i} one by one

Race condition: Between step 1 (check) and steps 3 (attach one by one), another connection could attach the same file or retrieve only partially attached databases. This result into errors

Spice query failed. Status: 400, body: Execution error: Failed to execute query.\nDuckDB connection failed.\nBinder Error: Unique file handle conflict: Cannot attach "attachment_SDFrVyz5_0" - the database file "/app/.spice/data/accelerated_duckdb.db" is already attached by database "attachment_23yXahuv_0"\nFor details, refer to the DuckDB manual: https://duckdb.org/docs/"}

2025-11-17T19:30:47.876652Z WARN datafusion_table_providers::sql::db_connection_pool::dbconnection::duckdbconn: my_table.duckdb not found among existing attachments

All DuckDB connections acquired from a single pool or created via try_clone() share the same catalog including attached databases, but not the search_path which is connection-level setting. PR updates attachment logic to attach databases once during pool initialization and use pre-computed search path for individual connections setup (`SET search_path = '...'] is a fast, idempotent session-level command)

┌─────────────────────────────────────────────────────────┐
│                  duckdb_database (self.db)              │
│  ┌─────────────────────────────────────────────────┐    │
│  │              Catalog (shared state)              │    │
│  │  - Tables                                        │    │
│  │  - Attached databases (attachment_xxx_0, etc.)   │    │
│  │  - Search paths (per-connection setting)         │    │
│  └─────────────────────────────────────────────────┘    │
│                                                         │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐   │
│  │ Connection 1 │  │ Connection 2 │  │ Connection 3 │   │
│  │  (original)  │  │ (try_clone)  │  │ (pool.get())  │   │
│  │              │  │              │  │              │   │
│  │ search_path: │  │ search_path: │  │ search_path: │   │
│  │   "main"     │  │   (default)  │  │   (default)  │   │
│  └──────────────┘  └──────────────┘  └──────────────┘   │
└─────────────────────────────────────────────────────────┘

@sgrebnov
Copy link
Collaborator Author

sgrebnov commented Dec 5, 2025

Closed in favor of #509

@sgrebnov sgrebnov closed this Dec 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants