deletion of .json files in _delta_log directory

[Curricane]

Environment

Delta-rs version: v0.23.0

Binding:

Environment:

• Cloud provider:
• OS:centos8
• Other:

---

Bug

We are encountering performance issues due to the large number of .json files generated in the _delta_log directory when using Delta Lake with delta-rs. These files, which represent transaction logs, have grown significantly over time in our setup.

The excessive number of small .json files is causing the following issues:

Increased overhead in file system operations (e.g., scanning and metadata retrieval).
Slower table initialization and query performance, especially when the transaction logs are not regularly cleaned.
Significant performance degradation in environments with object stores or distributed file systems (e.g., S3, HDFS, OSS).
We are looking for an efficient way to quickly delete or clean up these .json files to improve the overall performance of the Delta table.

I try to vaccum them by the code

async fn vacuum(delta_table: deltalake::DeltaTable) {
    let snapshot = delta_table.snapshot().unwrap();
    match VacuumBuilder::new(delta_table.log_store(), snapshot.clone())
        .with_retention_period(chrono::Duration::zero())
        .with_enforce_retention_duration(false)
        .with_commit_properties(CommitProperties::default().with_cleanup_expired_logs(Some(true)))
        .await
    {
        Ok((_, metrics)) => println!("vacuum metrics: {:?}", metrics),
        Err(e) => println!("vacuum error: {:?}", e),
    }
}

What happened:
.json still here
What you expected to happen:
delete some .json file if checkponit exists
How to reproduce it:

More details:

[ion-elgreco]

Json files get automatically removed in the post_commit_hook when they meet the interval and a checkpoint exists.

Can you create an MRE to showcase the behaviour you're seeing while using your table configuration?

[Curricane]

Json files get automatically removed in the post_commit_hook when they meet the interval and a checkpoint exists.

Can you create an MRE to showcase the behaviour you're seeing while using your table configuration?

Thank you for your response. I’m just starting to learn about delta-rs.
Could you let me know what the interval conditions are?

Meanwhile, is there a way for me to manually trigger the deletion? This would help me create an MRE.

[Curricane]

I find the way to delete .json.
First，when create table, set these metadata，like

    let mut table_properties = HashMap::new();
    table_properties.insert(
        TableProperty::LogRetentionDuration.as_ref().to_string(),
        Some("interval 2 minute".to_string()),
    );

    table_properties.insert(
        TableProperty::CheckpointInterval.as_ref().to_string(),
        Some("50".to_string()),
    );
    let delta_table = DeltaOps::try_from_uri(&table_path)
        .await?
        .create()
        .with_table_name(&static_view.name)
        .with_columns(struct_fields)
        .with_configuration(table_properties)
        .await?;
    Ok(Arc::new(delta_table))

Second, trigger clean when insert data

            let delta_table = open_table(&table_path).await?;

            let ops = DeltaOps(delta_table);
            let _ = ops
                .write(record_batches)
                .with_commit_properties(
                    CommitProperties::default()
                        .with_create_checkpoint(true)
                        .with_cleanup_expired_logs(Some(true)),
                )
                .await?;

Thus, when a checkpoint exists, during data insertion, .json files older than 2 minutes before the checkpoint time will be deleted.