The SDK sniffs a requested path to convert a dbfs:/-style path to the canonical /-style path expected and returned by the API. A path that contains the string dbfs: is not reachable from the SDK; /dbfs: (an actual path that users have created) is sent as a request with the path /. As an added bonus, this leads to infinite recursion when listing / returns /dbfs: as one of the directories.

The relevant code is at _DbfsPath.__init__(), and there is similar code at _VolumesPath.__init__()

    def __init__(self, api: files.DbfsAPI, src: str):
        self._path = pathlib.PurePosixPath(str(src).replace('dbfs:', '').replace('file:', ''))
        self._api = api

This malforms the path if dbfs: or file: appears anywhere in the path other than the beginning. A possible solution might use re.sub() instead of str.replace() to assert the beginning of the path in the replacement search.

import re

FILESYSTEM_PREFIXES = r'^(dbfs|file):'

# ...

    def __init__(self, api: files.DbfsAPI, src: str):
        self._path = pathlib.PurePosixPath(re.sub(FILESYSTEM_PREFIXES, '', str(src)))
        self._api = api

Other Information

• reported v0.40.0