Support az:// storage account name - #1798
Conversation
Deploying datachain with
|
| Latest commit: |
57e0260
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://cf50f345.datachain-2g6.pages.dev |
| Branch Preview URL: | https://az-account-name.datachain-2g6.pages.dev |
There was a problem hiding this comment.
Pull request overview
This PR updates the knowledge “bucket scan” tooling to correctly handle Azure Blob Storage URIs (az://) where the URI netloc is the container and the storage account name must be supplied separately to build HTTPS file links and to perform the anonymous-access probe.
Changes:
- Extend
source_to_https()to accept an explicitaccount_nameand returnNoneforaz://when it’s missing. - Add
--account-nametobucket_scan.py, defaulting toAZURE_STORAGE_ACCOUNT_NAME, and thread it through the Azure anonymous-access probe, listing (client_config), and link prefix generation. - Update unit tests to cover Azure link generation with/without an explicit account name.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
tests/unit/test_skill_knowledge_scripts.py |
Updates tests for Azure link prefix behavior with explicit account_name and None when missing. |
src/datachain/skill/knowledge/scripts/utils.py |
Changes source_to_https() to require account_name for az:// HTTPS prefix construction. |
src/datachain/skill/knowledge/scripts/bucket_scan.py |
Adds --account-name option and threads Azure account name into probing, listing configuration, and link prefix creation. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@dmpetrov I would like you to also take a look at these changes, what do you think? |
|
@dreadatour I've look breafly. How about an alternative approach #1859? |
Hm, sounds good to me! I haven't checked other products — my bad. Let me dive into the alternative approach 👍 |
|
Closing in favor of #1887 |
One of the Claude's findings along the way.
The storage account isn't part of an
az://URI (the netloc is the container), so file links and access probes must receive it explicitly. This threads the account name through the whole knowledge-skill scan workflow.source_to_https(): take an explicitaccount_name, falling back toAZURE_STORAGE_ACCOUNT_NAME; build the blob URL from account + container, and returnNoneforaz://when the account (or container) is missing rather than emitting a malformed link. The env fallback also gives dataset previews (schema.py) az links for free.bucket_scan: add--account-name(defaulting toAZURE_STORAGE_ACCOUNT_NAME) and thread it as a genericclient_configthrough the access probe and theread_storagelisting (so adlfs can resolve the account). The probe now honors the script's error contract: transport failures (e.g. a stale env account name) emit the error JSON and exit 1 instead of a traceback, and a probe timeout exits 124.bucket_overview(timeout fallback): same--account-nameflag, threaded into the fsspec listing, the bucket JSON, and the chain's session (Session.get(client_config=...)) so sampled File content stays readable during enrichment.SKILL.md: document--account-namefor all three Step-1 commands (thedatachain bucket statusCLI has no env default) and--anonfor the fallback when the access check reported anonymous.🤖 Generated with Claude Code