Writing a dataframe to a mounted s3 bucket in parquet format results in 0-byte files

[Taytay]

I've got an S3 bucket mounted. Doing simple touch and echo tests works like a champ. As does writing in other formats.

Then I tried to write a parquet file to the folder and noticed that it often yields 0-byte files with no error

# All of these work:
my_dataframe.to_csv(f"/mounted_folder/my_dataframe.csv")
my_dataframe.to_feather(f"/mounted_folder/my_dataframe.feather")
my_dataframe.to_parquet(f"./tmp/my_dataframe.parquet")

# But this often (not always) yields a zero-byte file:
my_dataframe.to_parquet(f"/mounted_folder/my_dataframe.parquet")

Seems like a bug in the way the flushing is working with the mounted filesystem, but pretty freaky...

[romilbhardwaj]

@Taytay thanks for the report! We've observed this behavior in different settings where certain disk operations result in a silent write failures. Our hypothesis is this is caused by our underlying mounting backend not supporting certain POSIX operations used under the hood by the .to_parquet method (e.g., appends don't work on mounted storage).

As a workaround, can you write the parquet file to a local path and then mv it to the mounted path?

[Taytay]

@romilbhardwaj : Thanks for the quick reply! Definitely can use that workaround - and was considering doing so - was just concerning to have a silent failure! It looks like the choice of S3 mounting techniques is being discussed in some other issues as well: #1160

[romilbhardwaj]

Absolutely, the silent failure is quite bad and should not happen. We'll keep this issue open till we fix our mounting backend

[concretevitamin]

A bunch of other users ran into this too. I think we should fix soon and/or add guides on workarounds to docs.

[Taytay]

For what it's worth, I ended up writing a function like this, so that the write happened outside of the fuse-mounted path, and then the copy moved it atomically into the fuse-mounted file-system:

def perform_operation_on_temp_file_and_then_move_to_destination_file(operation, actual_target_path):
    # Create a temporary file:
    temp_file = tempfile.NamedTemporaryFile(delete=False)
    temp_file.close()

    # Perform the operation on the temporary file:
    operation(temp_file.name)

    # Move the temporary file to the actual target path:
    shutil.move(temp_file.name, actual_target_path)

# Called like this:
perform_operation_on_temp_file_and_then_move_to_destination_file(lambda filename: train_prompts.to_parquet(filename, index=False), TRAIN_PROMPTS_FILENAME)

Hacky, but it worked okay.

[romilbhardwaj]

Hey @Taytay,

I'm trying to reproduce this bug with this minimal script I wrote. I ran this script for different dataframe sizes (100x100, 1000x1000, 5000x5000) on a mounted S3 bucket, and haven't been able to reproduce this issue over ~10000 runs.

Do you have any pointers on how I can reproduce this bug or any modifications I should make to my script?