Get access to the source information of a FileDataset

[SamVanhoutte]

Hello all,

I would like to get access to the source information of a FileDataset.
If you consider the following code:

ds = ws.datasets['myfiledataset']
print(str(ds))

Then there is an output like the following:

FileDataset
{
  "source": [
    "('datastorename', 'container/folder/')"
  ],
  "definition": [
    "GetDatastoreFiles"
  ],
  "registration": {
    "id": "66882<redacted>",
    "name": "dataset_name",
    "version": 1,
    "workspace": "Workspace.create(name='', subscription_id='', resource_group='')"
  }
}

Is there a documented & supported way to get access to that source information (the data store name and the folder on that location)? I want to reference the FileDataset and build a DataReferenceConfiguration object out of it (where I need the datastore_name and the path_on_datastore)

[MayMSFT]

Hi Sam

Are you asking whether there is api support to get the source info directly instead of parsing the json returned above?

Any reason you are converting the filedataset to datareference?

[SamVanhoutte]

Hey May,

I don't want to bother with file path & references in my scripts, etc. So, I like to use the FileDatasets that are defined in my Workspace, without having to bother where they reside.
When using an Estimator for training, I can just pass these datasets as inputs, like shown in the following code:

ds = 'myfiledataset'
input_ds = ws.datasets[ds].as_named_input(ds).as_mount(path_on_compute=ds)
estimator = TensorFlow(inputs = input_ds, **otherargs)

Since I'm now trying to use the same pattern (making a FileDataset available on the AmlCompute), but by using ScriptConfig and configured Environment, I don't seem to find another way documented that the way I was trying to take to create a DataReferenceConfiguration:

So, I would hope to get a supported way to get the values parsed by my_ugly_parsing_fx in the code below:

def my_ugly_parsing_fx(dataset: Dataset):
        import json
        j = json.loads(str(dataset).replace('FileDataset\n', ''))
        source = j['source'][0]
        sections = source.split("'")
        return sections[1], sections[3]

datastore, path = my_ugly_parsing_fx(ws.datasets[ds])
print(f'Adding mounting data reference for dataset {ds} on datastore {datastore} with path {path}')
dataref = DataReferenceConfiguration(datastore_name=datastore, path_on_datastore = path, path_on_compute = '/' + ds, mode = 'mount', overwrite = False)

scriptrunconfig.run_config.data_references = [datarefs]

So maybe the question should be more like: What is the best way to specify a DataReferenceConfiguration based on a DataSet?

[MayMSFT]

Hi Sam

You can pass in filedataset as argument in scriptrunconfig
from azureml.core import ScriptRunConfig

src = ScriptRunConfig(source_directory=script_folder, 
                      script='train_diabetes.py', 
                      # to mount the dataset on the remote compute and pass the mounted path as an argument to the training script
                      arguments =[dataset.as_named_input('diabetes').as_mount()])

sample notebook: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/work-with-data/datasets-tutorial/train-with-datasets/train-with-datasets.ipynb

[SamVanhoutte]

Thanks,
I have seen that yes, but I was hoping to be able to specify the folder location myself (like with the path_on_compute parameter in the DataReferenceConfiguration. Is that possible in some way?

[MayMSFT]

yes. you can pass path_on_compute to the as_mount() method on dataset
https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.dataset_consumption_config.datasetconsumptionconfig?view=azure-ml-py#as-mount-path-on-compute-none-

[SamVanhoutte]

Well, thanks! I learned a lot in this thread. So, my main problem is solved.
But the actual description of the issue (getting access to that FileDataset information, through the python SDK) is still worth being added , but the priority for me is definitely decreased now. Thx for being so helpful

[MayMSFT]

thanks for the feedback! glad that your issue has been resolved.