infer `use_fabric_endpoint` based on URI containing "dfs.fabric.microsoft.com"

[hutch3232]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I'm a polars user on Microsoft Fabric. I wasn't sure if this issue was better suited for polars or object-store. Decided maybe lower-level is better.

On recent versions of polars, in order to read from Fabric OneLake using the abfss:// URI, you have to pass storage_options={"use_fabric_endpoint": "true"} which I believe gets passed to some object-store credential provider.

It certainly is not a major challenge, but it feels untidy and redundant to have to specify that when I'm providing a URI that says it is fabric, e.g., "abfss://<workspace id>@onelake.dfs.fabric.microsoft.com/<lakehouse id>"

Describe the solution you'd like
I think whatever downstream effects this storage operation results in should have the default vary based on the URI passed. i.e., if it sees this URI, it should default to true.

Describe alternatives you've considered
An environment variable is also a possibility, and I tried a few to see if perhaps this already was configurable based on that, but it didn't seem to me. I tried, for example, USE_FABRIC_ENDPOINT.

Additional context
Possibly related issues:
#474
#430
pola-rs/polars#22096

I saw this was recently discussed in the MicrosoftFabric subreddit:
https://www.reddit.com/r/MicrosoftFabric/comments/1riuzs9/using_abfss_uris_with_new_polars_versions/

[kevinjqliu]

Thanks for reporting this! I agree this is a better experience.

I think this has already been addressed by #431
https://github.com/apache/arrow-rs-object-store/blame/0c380a13612a1c3d91aa9b2c263422b3049858c0/src/azure/builder.rs#L673-L678

and should already be in the v0.13.2 release
1cf4925

Could you give it another try and see if it works automatically?

[alamb]

CLosing as this seems to have been fixed already. Thanks @kevinjqliu and @hutch3232

[hutch3232]

It took me awhile to get to this, but this isn't working for me. I'm using the latest polars 1.43.0 which uses object-store 0.13.2 which includes this fix.

However, when I read paths (scan_csv) with an abfss path as seen in the issue description, polars will error out. It basically complains that there are missing required args tenant_id, client_id, and token_file_path. However if I pass use_fabric_endpoint as true to the storage_options then it works. So that led me to believe for some reason this is still not being inferred from the path. Quite possibly a polars issue, but wanted to raise here since I already have this issue.