CHECKPOINT errors during concurrent MERGE and INSERT writes

[dmcfarland]

Hello,

I'm encountering transaction conflicts and duplicate key errors when trying to run CHECKPOINT on a Ducklake instance that is under a continuous write load from two separate processes.

Environment

DuckDB Version: 1.4.0
Storage: S3
Catalog Database: Postgres

Scenario

I have two independent, long-running processes:
Process 1 (Upserts): Continuously uses the MERGE INTO statement to update existing rows or insert new ones.
Process 2 (Appends): Continuously uses the INSERT INTO statement to append new rows.

While these two processes are running, I attempt to run CHECKPOINT from a third separate connection.

Problem

When I execute the CHECKPOINT command, the operation fails with one of two errors.

Initially, I received a duplicate key violation on an internal metadata table:

-- First attempt
CHECKPOINT;

-- ERROR OUTPUT:
-- TransactionContext Error: Failed to perform CHECKPOINT; in DuckLake:
-- Failed to commit: Failed to commit DuckLake transaction.
-- Exceeded the maximum retry count of 10 set by the ducklake_max_retry_count setting.
-- ...
-- Failed to write data file information to DuckLake: Failed to copy data:
-- ERROR:  duplicate key value violates unique constraint "ducklake_data_file_pkey"
-- DETAIL:  Key (data_file_id)=(4755) already exists.
-- CONTEXT:  COPY ducklake_data_file, line 1

After increasing the retry count as suggested (SET ducklake_max_retry_count = 100;), I then encountered a transaction conflict error:

-- Second attempt after increasing retry count
CHECKPOINT;

-- ERROR OUTPUT:
-- TransactionContext Error: Failed to perform CHECKPOINT; in DuckLake:
-- Failed to commit: Failed to commit DuckLake transaction.
-- Transaction conflict - attempting to compact table with index "99" - but another transaction has deleted from it

Questions

• Is it required to pause all write operations (MERGE, INSERT) before executing a CHECKPOINT to guarantee it completes successfully?
• Should I be concerned about metadata corruption after seeing these errors?
• What is the recommended strategy for performing maintenance operations like CHECKPOINT on a database with continuous, high-frequency concurrent writes?