Is it possible to retrieve metadata associated with the blobs using Datastore?

[mokarian]

We have some images in Azure storage account. All those images (blobs) have metadata .
calling download we can download the files from the provided path but we are missing the metadata associated to those files.
In fact metadata of the files contains important information on each image (e.g. tags) that we need them in our ML pipeline.
Is there any way to keep metadata of a blob storage when we download?
E.g.:

blob_datastore = Datastore.register_azure_blob_container(workspace=aml_workspace,
                                                             datastore_name=blob_datastore_name,
                                                             container_name=container_name,
                                                             account_name=account_name,
                                                             account_key=account_key)

def_blob_store = Datastore(aml_workspace, "train")

datastore_path = [
       DataPath(def_blob_store, '*.jpeg')]

file_dataset = Dataset.File.from_files(path=datastore_path)
file_dataset.download("files")

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: b080f459-b191-a60e-e327-720ef095165c
• Version Independent ID: 53d1b412-0c47-1044-3c6c-7c8c0f1eb0c0
• Content: azureml.data.FileDataset class - Azure Machine Learning Python
• Content Source: AzureML-Docset/stable/docs-ref-autogen/azureml-core/azureml.data.FileDataset.yml
• Service: machine-learning
• Sub-service: core
• GitHub Login: @DebFro
• Microsoft Alias: debfro

[MayMSFT]

@mokarian thanks for the question.

We currently dont support retrieve meta data of the files via dataset SDK. I will convert your question into feature ask and add to our product backlog. To better understand your scenario: what kind of meta data are you looking for? and how will you use these meta data in your script?

[mokarian]

Thank you @MayMSFT for your response. Here is our scenario.
We have numerous images located in a container in a storage account. Each of those images have some tags associated with them. Those tags defines some information that helps us in our ML project. For instance we have tags for images that helps us to assign them when we upload images to Custom Vision for classification.
At this time we store those tags as metadata in blobs and in each stage of our ML pipeline we access them using BlobServiceClient.
We were considering to use Datastore to access those images but it seems that we will miss those metadata (that we will need in several stages of our ML project). Hope this makes sense.

[meyetman]

Hi @mokarian thanks for the additional information! We are adding that to the feature ask. We will close this issue for now, but feel free to reopen it if you have any additional information or requests. #please-close

[lostmygithubaccount]

hi @mokarian are you looking for metadata like this?

I would recommend using ADLFS, a simple implementation of a Pythonic filesystem on top of Blob or ADLS. You can easily run fs.info as in the screenshot above to retrieve metadata

various examples in https://github.com/Azure/azureml-examples (particularly the dask and xgboost tutorials) demonstrate using ADLFS