Keep HDFS connection alive

[hamidgh09]

Hi,
I recently noticed a large amount of delay due to the HDFS connection establishment latency.
Looks like for every operation I do, a new connection is created and disconnects when it's done.
For instance the following code snippet results in two separate connections:

import pandas as pd
import subprocess
from deltalake import write_deltalake, DeltaTable

table_path = "hdfs://my.hdfs.server.address/delta_table_test"

# -------------------------------
# Step 1: Create initial large DataFrame and write it to HDFS
# -------------------------------
initial_df = pd.DataFrame({
    "id": range(1_000_000),
    "value": ["initial_data"] * 1_000_000
})

write_deltalake(table_path, initial_df, mode="overwrite")
print("Delta table created on HDFS with initial data.")

# -------------------------------
# Step 2: Append additional data to the Delta table
# -------------------------------
append_df = pd.DataFrame({
    "id": range(1_000_000, 1_500_000),
    "value": ["append_data"] * 500_000
})

write_deltalake(table_path, append_df, mode="append")
print("Appended new data to Delta table.")

Is there any way that I can keep the connection open and re-use it in the script?
I assume it should be the same if I use different filesystems like Azure, AWS, etc. correct?

[rtyler]

Can you try creating a DeltaTable first from your table_path and then pass that into write_deltalake?

dt = DeltaTable(table_path)
# do some work
write_deltalake(dt, append_df, mode='append')
# do some more work
write_deltalake(dt, append_df, mode='append')

[hamidgh09]

ah, yes! It keeps the connection open as long as I have the dt. Thanks.