Interested to see some benchmark of objstore with PyTorch DataLoader

[tchaton]

Hey @kylebarron.

I am the main author of LitData: Lightning-AI/litData and we have lot of GeoSpatial users.

I am quite curious to see how objstore will behave with PyTorch DataLoader.

We have tried to build a rust backend to pull the data (our own or tried hf_transfer from Hugging Face).

It was faster in raw benchmark outside of PyTorch DataLoader but it became 10-30% slower otherwise. The CPU usage was pretty chaotic using a rust backend and constantly at 100% with boto3.

I would be pretty keen to connect and see if there is room for collaboration. I still believe that we could make a rush backend works well.

Best regards,
Thomas Chaton.

[kylebarron]

I haven't used PyTorch and I'm not familiar with its DataLoader. Is it async-compatible? The greatest performance improvements from obstore come when using it with high concurrency in async Python code.