[Bug]: Failed to commit transaction when branch is specified

[BinEricCoder]

What happened

I am using DuckDB to interact with a Nessie catalog using the Iceberg REST API.
In the documentation for Nessie, it states that an insert operation can be done into a branch by specifiying the branch name after the table name as follows:

https://projectnessie.org/guides/iceberg-rest/#time-travel
INSERTing or UPDATEing data works similarly:
INSERT INTO nessie.my_namespace.my_table@my_other_branch ( id, val ) VALUES ( 123, 'some value' );

I am using DuckDB to do this, and it is failing, from what it seems to be Nessie's side.

How to reproduce it

Start a DuckDB CLI instances, Nessie and MinIO running in docker containers.

duckdb

Install and load Iceberg

INSTALL iceberg;
LOAD iceberg;

Create MinIO credentials and endpoints

CREATE OR REPLACE SECRET minio_s3 (
     TYPE S3,
     KEY_ID 'admin', -- Replace with your MinIO Access Key
     SECRET 'password', -- Replace with your MinIO Secret Key
     ENDPOINT 'minio-0:9301', -- Replace with your MinIO host and S3 API port
     USE_SSL false, -- Use 'true' if you have SSL configured (default is false for local MinIO)
     URL_STYLE 'path' -- Required for path-style access used by MinIO
 );

Create Iceberg auth token

CREATE OR REPLACE SECRET iceberg_rest_secret (
    TYPE ICEBERG,
     TOKEN 'dummy' -- Use 'dummy' if no specific token is needed or replace with an actual Bearer token/Client Secret
);

Attach and use default warehouse="warehouse"

ATTACH 'warehouse' AS db (
     TYPE ICEBERG,
     ENDPOINT 'http://nessie:19120/iceberg', -- Replace with your Catalog host and port
     SECRET iceberg_rest_secret, -- Reference the secret created in step 2 (optional, if no auth is needed)
     -- Important: If your catalog vends credentials (default behavior), and you want
     -- DuckDB to use your MinIO secret instead, add this parameter:
     ACCESS_DELEGATION_MODE 'none'
);

Selecting from a table and explicitly defining the branch works:

CALL truncate_duckdb_logs();
CALL enable_logging('HTTP');
SELECT * FROM db.public."simple_table@main";
SELECT request.type, request.url, response.status FROM duckdb_logs_parsed('HTTP');

Output from the http logs below, showing no failure

┌─────────┬───────────────────────────────────────────────────────────────────────────────────────────────┬────────────────────┐
│  type   │                                              url                                              │       status       │
│ varchar │                                            varchar                                            │      varchar       │
├─────────┼───────────────────────────────────────────────────────────────────────────────────────────────┼────────────────────┤
│ HEAD    │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public                             │ NoContent_204      │
│ HEAD    │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public/tables/simple_table%40main  │ NoContent_204      │
│ GET     │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public/tables/simple_table%40main  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ OK_200             │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ PartialContent_206 │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ PartialContent_206 │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ PartialContent_206 │
│ GET     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data…  │ PartialContent_206 │
├─────────┴───────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┤
│ 12 rows                                                                                                            3 columns │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

INSERT INTO db.public."simple_table@main" ( col1, col2 ) VALUES ( 123, 'inserted into dev first' );

CALL truncate_duckdb_logs();
CALL enable_logging('HTTP');
INSERT INTO db.public."simple_table@main" ( col1, col2 ) VALUES ( 123, 'inserted into dev first' );
SELECT request.type, request.url, response.status FROM duckdb_logs_parsed('HTTP');

Output from http log shown below, indicating failure on commit

┌─────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────┐
│  type   │                                                url                                                 │    status     │
│ varchar │                                              varchar                                               │    varchar    │
├─────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────┤
│ HEAD    │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public                                  │ NoContent_204 │
│ HEAD    │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public/tables/simple_table%40main       │ NoContent_204 │
│ GET     │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/namespaces/public/tables/simple_table%40main       │ OK_200        │
│ PUT     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data/019b…  │ OK_200        │
│ PUT     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data/a069…  │ OK_200        │
│ HEAD    │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data/a069…  │ OK_200        │
│ PUT     │ http://minio-0:9301/my-bucket/public/simple_table_0bc3949a-e475-4ff5-b823-954d1628eb81/data/snap…  │ OK_200        │
│ POST    │ http://nessie:19120/iceberg/v1/main%7Cwarehouse/transactions/commit                                │ NotFound_404  │
└─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────┘

Nessie server type (docker/uber-jar/built from source) and version

Docke 0.106.1

Client type (Ex: UI/Spark/pynessie ...) and version

DuckDB CLI v1.4.3

Additional information

No response

[dimas-b]

@BinEricCoder : would you be able to provide actual error messages reported by Nessie? I was only able to find HTTP status codes in your issue description, which are hardly sufficient to understand the underlying problem 😅

[BinEricCoder]

@dimas-b well noted, please see output for the query in question below

INSERT INTO db.public."simple_table@main" ( col1, col2 ) VALUES ( 123, 'inserted into dev first' );

TransactionContext Error:
Failed to commit: Failed to commit Iceberg transaction: Request to 'http://nessie:19120/iceberg/v1/main%7Cwarehouse/transactions/commit' returned a non-200 status code (NotFound_404), with reason: Not Found, body: {
  "error" : {
    "message" : "UnknownEntity does not exist: public.simple_table@main",
    "type" : "NotFoundException",
    "code" : 404
  }
}

[dimas-b]

Seeing @main in the table name in the error message I'd guess the request was not encoded properly. This part would normally be interpreted on the client side (e.g. when using Spark).

Note the main%7C fragment in the URI - this is the branch name which the Nessie Server would use for committing the change (this part is the correct branch specification).

[BinEricCoder]

I can MITM the requests going to Nessie if that would help to see what is in the POST request to commit, all other requests seem to be fine.

For additional context, I explicitly set the branch to main in the SELECT and INSERT INTO requests because I want to use other branches eventually using Iceberg REST type catalog on Nessie, but wanted to eliminate it being an issue on a non-main branch, for which some default url paths exist.

[dimas-b]

@BinEricCoder : you use case makes sense! I believe it works in Spark.

I think the question is whether it works correctly in your particular engine... I never played with DuckDB + Nessie personally, so I do not really know that first hand 😅

If you have low-level request dumps, I can certainly help and confirm whether they are valid from the Nessie API perspective.