What happens?
When several clients run a bulk CREATE OR REPLACE TABLE ... AS SELECT * FROM read_parquet(...) in parallel against a persistent (on-disk) database served via quack_serve, the server process dies mid-transfer. All clients fail at the same time with:
IO Error: Failed to send message: IO Error: Could not connect to server error for HTTP POST to 'http://localhost:9494/quack'
The server process is still listed as running in the OS task manager, but it no longer serves requests.
Restarting the same parallel insert does not recover: every client now fails immediately with a malformed/garbled error referencing the target table, e.g.:
Schema\0main\0main\0Table\0main\0BigTable1
(each client reports the same with its own target table name: BigTable2, BigTable3, …). A full server restart is required to make the endpoint usable again.
The failure is concurrency-dependent: with a lower number of parallel clients the same workload completes successfully. In my tests up to 7 clients in parallel finished without issues; the crash starts to appear at higher concurrency (reproduced here with 10 clients).
To Reproduce
Server (persistent, on-disk DuckDB database):
CALL quack_serve('quack:localhost:9494', token = 'super_secret');
Then start N clients in parallel (e.g. N = 10), each running:
CREATE SECRET (TYPE quack, TOKEN 'super_secret');
ATTACH 'quack:localhost:9494' AS remote;
CREATE OR REPLACE TABLE remote.BigTable<k> AS
SELECT * FROM read_parquet('data.parquet') LIMIT 4000000;
where <k> is a per-client index (so each client writes to a distinct table) and data.parquet is a ~60 MB Parquet file (~3.7M rows). Any reasonably large Parquet file should reproduce it.
Observed:
- First run — all clients run for ~90–100 s, then fail together with
Could not connect to server. The server process stays alive but unresponsive.
- Second run (server not restarted) — all clients fail almost immediately (~37–41 s) with the garbled
Schema\0main\0main\0Table\0main\0BigTable<k> message.
A single client (N = 1) completes successfully, and so does a moderate level of concurrency (up to ~7 clients in my tests); the failure shows up only at higher concurrency.
Environment
|
|
| OS |
Windows 11 (x64) |
| CPU |
Intel Core i5-1135G7 @ 2.40 GHz (4 cores / 8 threads) |
| RAM |
16 GB |
| Setup |
DuckDB CLI + quack extension |
| DuckDB version |
v1.5.3 (Variegata) |
What happens?
When several clients run a bulk
CREATE OR REPLACE TABLE ... AS SELECT * FROM read_parquet(...)in parallel against a persistent (on-disk) database served viaquack_serve, the server process dies mid-transfer. All clients fail at the same time with:The server process is still listed as running in the OS task manager, but it no longer serves requests.
Restarting the same parallel insert does not recover: every client now fails immediately with a malformed/garbled error referencing the target table, e.g.:
(each client reports the same with its own target table name:
BigTable2,BigTable3, …). A full server restart is required to make the endpoint usable again.The failure is concurrency-dependent: with a lower number of parallel clients the same workload completes successfully. In my tests up to 7 clients in parallel finished without issues; the crash starts to appear at higher concurrency (reproduced here with 10 clients).
To Reproduce
Server (persistent, on-disk DuckDB database):
Then start N clients in parallel (e.g.
N = 10), each running:where
<k>is a per-client index (so each client writes to a distinct table) anddata.parquetis a ~60 MB Parquet file (~3.7M rows). Any reasonably large Parquet file should reproduce it.Observed:
Could not connect to server. The server process stays alive but unresponsive.Schema\0main\0main\0Table\0main\0BigTable<k>message.A single client (
N = 1) completes successfully, and so does a moderate level of concurrency (up to ~7 clients in my tests); the failure shows up only at higher concurrency.Environment