Insertion order difference between 0.15.1 and 0.24.0

[kobi4000]

I have been using delta-rs to accumulate timeserieses.
Recently I've updated delta-rs from 0.15.1 to 0.24.0, and I've found a new behaviour as to the insertion order.

I've generated a sample table with

for i in range(3):
    now = datetime.now()
    df = pd.DataFrame({
        "datetime": [now, now+timedelta(microseconds=1)],
        "num": [8, 9],
        "letter": ["dd", "ee"]
    }) 
    write_deltalake( 
        table_path,
        df,
        mode="append",
    ) 
    time.sleep(2) 

inserting three timeseries (with its timestamp column in ascending order) to simulate my use case, in different transaction.

Below is the content of actions table and data for 0.15.1 and 0.24.0, respectively.

Output is the print out of

dt.get_add_actions(flatten=True).to_pandas()[['path','modification_time']]
dt.to_pandas()

With 0.15.1,

deltalake.__version__='0.15.1'
                                               path       modification_time
0  0-d1ab7ec2-0bab-4f4c-a10d-8f163d7c0339-0.parquet 2025-02-17 14:48:20.690
1  1-86a83904-8a5b-4a57-8d9c-6fceb5211838-0.parquet 2025-02-17 14:48:22.712
2  2-93ffa290-bd94-48ec-a946-33f4848b54c1-0.parquet 2025-02-17 14:48:24.728
                    datetime  num letter
0 2025-02-17 23:48:20.677320    8     dd
1 2025-02-17 23:48:20.677321    9     ee
2 2025-02-17 23:48:22.697655    8     dd
3 2025-02-17 23:48:22.697656    9     ee
4 2025-02-17 23:48:24.715909    8     dd
5 2025-02-17 23:48:24.715910    9     ee

Data is readily usable/scannable for my use case as it's timestmap is already in asending order.

But, with 0.24.0,

deltalake.__version__='0.24.0'
                                                path       modification_time
0  part-00001-fc889c00-c188-44b8-8850-81a337b95cf... 2025-02-17 14:45:42.336
1  part-00001-2530a22d-2171-4c5e-a916-9b427b2c000... 2025-02-17 14:45:40.322
2  part-00001-e9d77e48-3ca9-4eee-aaf9-0cd12d2f6ed... 2025-02-17 14:45:38.308
                    datetime  num letter
0 2025-02-17 23:45:42.330398    8     dd
1 2025-02-17 23:45:42.330399    9     ee
2 2025-02-17 23:45:40.312910    8     dd
3 2025-02-17 23:45:40.312911    9     ee
4 2025-02-17 23:45:38.295785    8     dd
5 2025-02-17 23:45:38.295786    9     ee

Timestamp across parquet files are descending order while data in same transaction (same parauet file) in asending order.

It seems the output data is aggregated in the order apperaed in the add action table.
Is there any way I can control this aggregation/scan order, so the insertion/trasnaction order (adding new data to bottom) is preserved?

This is mentioned also in #2659 (comment)

[ion-elgreco]

Delta lake doesn't guarantee insertion order