post_commithook_properties checkpoint flag is not honored

[havrylenkok]

Environment

Delta-rs version: 1.1.4

Binding:

Environment:

• Cloud provider: AWS
• OS:
• Other: python 3.11, table is written to S3 with DDB locking

---

Bug

What happened:

write_deltalake(
    dt,
    arrow_table,
    mode="append",
    schema_mode=None,
    storage_options=storage_options,
    writer_properties=writer_properties,
    post_commithook_properties=PostCommitHookProperties(create_checkpoint=False),
    partition_by=["partition_date"],
)

creates checkpoints every 100 commits

What you expected to happen:

no automatic checkpoints are created

How to reproduce it:

run the snippet to write to the table and create 101 commits

More details:

docs on these https://delta-io.github.io/delta-rs/api/transaction/#deltalake.PostCommitHookProperties

Reading the table created with deltalake in S3 via Athena often doesn't work because of disagreement on contents of last checkpoint file in this library and trino trinodb/trino#18760

I'm trying to disable any kind of automatic checkpoint creation so then when I create them manually via table.create_checkpoint I fix the file in S3 immediately and do not break queries.

[fvaleye]

Hi @havrylenkok 👋,
Thank you for the report.
Could you please provide a code snippet that reproduces the error you’re experiencing?

[havrylenkok]

@fvaleye I tried to reproduce it in a simple script from scratch and couldn't. Neither when creating a table in local fs, nor in S3 with ddb locking and code close to what I'm using in my app.

Sorry for a not reproducible report, should be fine to close.

For posterity: in my app I can reliably see it creating checkpoints for commits 99 and 199. However after that the table, as I expected it to, continues for hundreds of commits without new checkpoints. While I reported it originally for write_deltalake, digging deeper into the logs, those first two checkpoints get created at some point in the middle of looping over created partitions and running table.optimize.z_order and table.optimize.compact with partition_filters.

[fvaleye]

@fvaleye I tried to reproduce it in a simple script from scratch and couldn't. Neither when creating a table in local fs, nor in S3 with ddb locking and code close to what I'm using in my app.

Sorry for a not reproducible report, should be fine to close.

For posterity: in my app I can reliably see it creating checkpoints for commits 99 and 199. However after that the table, as I expected it to, continues for hundreds of commits without new checkpoints. While I reported it originally for write_deltalake, digging deeper into the logs, those first two checkpoints get created at some point in the middle of looping over created partitions and running table.optimize.z_order and table.optimize.compact with partition_filters.

Thanks @havrylenkok for trying!
I will close this issue for now. Please feel free to re-open it if you can reproduce the problem.