Cannot read ADLS2 table with delta rs > 1.1.0 using `az://` prefix

[joepatol]

Environment

Delta-rs version: 1.1.2 and 1.1.3

Binding: Python

Environment:

• Cloud provider: Azure
• OS: Linux
• Other:

---

Bug

Since upgrading beyond version 1.1.0 my code that reads a deltatable from Azure adls gen 2 stopped working
What happened:
I received the message _internal.TableNotFoundError: no log files
What you expected to happen:
To get a table
How to reproduce it:

from azure.identity import AzureCliCredential
from deltalake import DeltaTable

storage_options = {
    "account_name": "myaccount",
    "container_name": "my-container",
    "token": AzureCliCredential().get_token("https://storage.azure.com/.default").token,
}

dt = DeltaTable("az://my-container/my_table", storage_options=storage_options)
print(dt.to_pandas().head())

Version <= 1.1.0 this work fine

With > 1.1.0

_internal.TableNotFoundError: no log files

More details:
It seems to be Azure specific, if I store the table on my local machine it works fine

[rtyler]

I believe I saw this manifest in our integration tests but was not able to nail it down entirely. Can you try changing the url from az:// to abfs:// and see how that treats you?

[joepatol]

@rtyler that indeed fixes it

[rtyler]

Thanks for confirming @joepatol! There is something fishy that has started to happen as of object_store 0.12.x and our codebase where it appears that when using the az:// URL syntax the container name is getting somewhere along the lines when creating URLs in delta-rs.

I'm going to leave this open since it's still a bug worth fixing, but I'm glad you have a workaroudn

[joepatol]

@rtyler thx, not sure if it's still the case but this comes from the docs, maybe at least change it there so people are pointed in the right direction

[ion-elgreco]

This might be just related to an upstream change in object-store: apache/arrow-rs-object-store#443

[ion-elgreco]

@joepatol Can you please confirm which versions it didn't work?

Your description says 1.1.2 and 1.1.3 but then you also mention since 1.1+ this is quite a difference because 1.1.0 had object_store 0.12.2 while the others are on object_store 0.12.3, so I would like to know exactly to remove any confusion

[joepatol]

Sorry for the confusion. It does work on 1.1.0 but not on 1.1.2 and 1.1.3

[rtyler]

@ion-elgreco I think the "fix" for us here is probably going to just be to drop az:// support. I don't think it's worth our time to investigate the object_store issue when using abfss:// solves this.

[ion-elgreco]

Depends a bit, whether upstream it will be fixed or not. Or its breaking because they solved an issue?