[Bug]: dbt fails to create a directory tree due to lack of rights on an already existing parent directory

[ArnaudValmary]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When running dbt run, the adapter try to create a directory tree but fails due to lack of rights (HTTP error 403).
Example :

• Dremio space : my_space
• Dremio folder to create : my_dir.sub_dir_1.sub_dir_2

dbt-dremio try to create successively:

• my_dir
• my_dir.sub_dir1
• my_dir.sub_dir_1.sub_dir_2

If my_dir and my_dir.sub_dir_1 already exist with good write rights, this is no problem. The HTTP error is 400 and the exception DremioAlreadyExistsException is ignored. The process continue (function dbt/adapters/dremio/connections:py_create_folders)

But, if my_dir already exist without good write rights, and my_dir.sub_dir_1 already exist with good write rights, there is a problem. The HTTP error is 403 and the exception DremioPermissionException is catched. The process stops immediately.

Expected Behavior

I think, it's necessary to ignore DremioPermissionException like DremioAlreadyExistsException in function dbt/adapters/dremio/connections:py_create_folders by adding :

            except DremioPermissionException:
                logger.debug(f"Folder {folder} no permission.")

Steps To Reproduce

1. Create manually a folder tree with a minimum of two folders in a Dremio space or from a previous run
2. Remove write rights on the first folder
3. Configure a dbt project to create a new sub directory in the manually created second folder
4. Run dbt run.
5. Observe the failure with DremioPermissionException.

Environment

- OS: macOS Tahoe 26.3.1 (a)
- dbt-dremio: 1.10.0
- dbt-core: 1.10
- Dremio Software: 25.2.17
- Dremio Cloud: N/A

Relevant log output

08:59:36  dremio adapter: Creating folder(s): my_space_my_dir.sub_dir_1.sub_dir_2
08:59:37  On create_my_space_my_dir.sub_dir_1.sub_dir_2: Close
08:59:37  Connection 'master' was properly closed.
08:59:37  Connection 'create_my_space_my_dir.sub_dir_1.sub_dir_2' was properly closed.
08:59:37  
08:59:37  Finished running  in 0 hours 0 minutes and 4.31 seconds (4.31s).
08:59:37  Encountered an error:
No permission:: (403 Client Error: Forbidden for url: https://dremio-pf-cloud.staging.echonet:443/api/v3/catalog): ({"errorMessage":"Permission Denied","moreInfo":""})
08:59:37  Traceback (most recent call last):
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 182, in wrapper
    result, success = func(*args, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 128, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 276, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 321, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 391, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 368, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/requires.py", line 408, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/cli/main.py", line 587, in run
    results = task.run()
              ^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 626, in run
    result = self.execute_with_hooks(selected_uids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 559, in execute_with_hooks
    before_run_status = self.before_run(adapter, selected_uids)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/task/run.py", line 1124, in before_run
    self.create_schemas(adapter, required_schemas)
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 755, in create_schemas
    create_future.result()
  File "/opt/homebrew/Cellar/python@3.11/3.11.15/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.15/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/python@3.11/3.11.15/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt_common/utils/executor.py", line 16, in connected
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/task/runnable.py", line 717, in create_schema
    adapter.create_schema(relation)
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/impl.py", line 96, in create_schema
    self.connections.create_catalog(relation)
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/connections.py", line 271, in create_catalog
    self._create_folders(database, schema, rest_client)
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/connections.py", line 441, in _create_folders
    rest_client.create_catalog_api(folder_json)
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/api/rest/client.py", line 101, in create_catalog_api
    return _post(
           ^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/api/rest/utils.py", line 67, in _post
    return _check_error(response, details)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myuser/Library/Caches/pypoetry/virtualenvs/dbt-dremio-project-EE1GxAML-py3.11/lib/python3.11/site-packages/dbt/adapters/dremio/api/rest/utils.py", line 133, in _check_error
    raise DremioPermissionException("No permission:" + details, error,
dbt.adapters.dremio.api.rest.error.DremioPermissionException: No permission:: (403 Client Error: Forbidden for url: https://dremio-pf-cloud.staging.echonet:443/api/v3/catalog): ({"errorMessage":"Permission Denied","moreInfo":""})

[bapBardas]

We've faced the same issue and v1.10.1 does fix it.