Documentation for Azure DataLake2 doesn't seem to match what actually happens

[pamelafox]

• Package Name: azure-storage-file-datalake
• Package Version: 12.14.0
• Operating System: Mac/Linux
• Python Version: 3.11

Describe the bug

We have code like this:

    user_blob_container_client: FileSystemClient = current_app.config[CONFIG_USER_BLOB_CONTAINER_CLIENT]
    user_directory_client = user_blob_container_client.get_directory_client(user_oid)
    await user_directory_client.set_access_control(owner=user_oid)
    file_client = user_directory_client.get_file_client(file.filename)

The documentation for both get_directory_client and get_file_client claim that the directory/file need not exist.

However, I get this error on get_file_client:

Traceback (most recent call last):
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/quart/app.py", line 1376, in handle_request
    return await self.full_dispatch_request(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/quart/app.py", line 1414, in full_dispatch_request
    result = await self.handle_user_exception(error)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/quart/app.py", line 1007, in handle_user_exception
    raise error
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/quart/app.py", line 1412, in full_dispatch_request
    result = await self.dispatch_request(request_context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/quart/app.py", line 1506, in dispatch_request
    return await self.ensure_async(handler)(**request_.view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/app/backend/decorators.py", line 53, in auth_handler
    return await route_fn(auth_claims)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/app/backend/app.py", line 248, in upload
    await user_directory_client.set_access_control(owner=user_oid)
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/core/tracing/decorator_async.py", line 88, in wrapper_use_tracer
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/storage/filedatalake/aio/_path_client_async.py", line 334, in set_access_control
    process_storage_error(error)
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/storage/filedatalake/_deserialize.py", line 221, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/storage/filedatalake/aio/_path_client_async.py", line 332, in set_access_control
    return await self._client.path.set_access_control(**options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/core/tracing/decorator_async.py", line 88, in wrapper_use_tracer
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/storage/filedatalake/_generated/aio/operations/_path_operations.py", line 1313, in set_access_control
    map_error(status_code=response.status_code, response=response, error_map=error_map)
  File "/workspaces/azure-search-openai-demo/.venv/lib/python3.11/site-packages/azure/core/exceptions.py", line 164, in map_error
    raise error
azure.core.exceptions.ResourceNotFoundError: The specified path does not exist.
RequestId:fc70a517-c01f-0035-13a5-7ffc1c000000
Time:2024-03-26T17:44:51.3779940Z
ErrorCode:PathNotFound

So that implies to me that the documentation is wrong, and that I cannot assume the get_*_client functions will create the directory/file for me. As a workaround, I will try to explicitly create them first.

To Reproduce
Steps to reproduce the behavior:

1. You could deploy Optional upload documents feature Azure-Samples/azure-search-openai-demo#1395 but it'd be a lot of work. You'd need to set up an ADLS2 storage container and try the get_directory_client on a directory that didn't exist and then get_file_client on a filename that didn't exist.

[pamelafox]

eek, sorry, mis-clicked submit, still drafting.

[pamelafox]

Updated the description.

[jalauzon-msft]

Hi @pamelafox Pamela, you are correct that get_*_client will not create the resource for you and so you must explicitly create it before performing any operations. We also offer .exists() methods to check if the resource exists or not. The documentation is meant to indicate that the resource need not exist for you to create a client object, since making a client does not implicitly perform any network operations on the resource. Hopefully that helps clear it up, thanks.

[pamelafox]

Thanks! It seems like the documention could be reworded? I could make the PR but am not certain of the right description. Perhaps it could also explicitly mention that we should use exists() for checking existence, as I ended up calling user_directory_client.get_directory_properties(), but I'd like to do whatever is most efficient/recommended.