[Bug]: 1.5.1 regression : Onelake broken

[djouallah]

What happened?

read write are broken on azure, i think it was a change of behavieur in object store, I report it in delta kernel as duckdb delta is broken too

delta-io/delta-kernel-rs#2433

Expected behavior

read and write to azure you will get this error

IOException: IO Error: DeltaKernel GenericError (5): Generic delta kernel error: No files in log segment

Operating System

Linux

Binding

Python

Bindings Version

1.5.1

Steps to reproduce

No response

Relevant logs

[rtyler]

Thanks for the bug report. Was this using the AZ file scheme or just any Azure object store URL?

[djouallah]

we use abfss url mainly

[rtyler]

Thanks @djouallah , I sent you a message in the Delta slack in case that's an easier way to pass some debug logs back and forth 😄

I'm annoyed that it doesn't seem like our integration tests with Azure caught whatever failure you're seeing. My hunch is that there's some path creation that's no longer working correctly because of changes in delta-kernel-rs or perhaps even object_store 0.13 where there has been some chanegs to Azure lately too

[wernerdaehn]

I concur with the finding. I do a simple write_table using the abfss file path into an empty directory and with mode=overwrite, and got the same error. The write is partitioned by one column.
What I noticed is that the parquet file is created, the delta log directory also but it remains empty.
changing the requirements.txt to deltalake==1.5.0 and all is back to working.

[rtyler]

@wernerdaehn is the behavior you are seeing specifically with OneLake or with some other flavor of Azure storage?

[wernerdaehn]

Onelake.

[rtyler]

I am watching the upstream issue for the object_store crate that we depend on. Thanks again for the early report and triage @djouallah!