bug: Databricks Backend: Specifying a non-existent volume in a volume in the 3-level namespace of Unity Catalog

[akuwano]

What happened?

Description

When connecting to Databricks with Unity Catalog enabled, the backend appears to be trying to access a memtable volume that does not exist.

Error Message

ServerOperationError: [UC_VOLUME_NOT_FOUND] Volume `my_catalogmy_schemausername-py=3131-pid=8608` does not exist. 
Please use 'SHOW VOLUMES' to list available volumes. SQLSTATE: 42704

Expected Behavior

The volume name should be generated as:

my_catalog.my_schema.<volume_name>

Actual Behavior

The volume name is generated without dots:

my_catalogmy_schemausername-py=3131-pid=8608

Steps to Reproduce

import ibis
import os

engine = ibis.databricks.connect(
    server_hostname=os.environ.get("DATABRICKS_SERVER_HOSTNAME"),
    http_path=os.environ.get("DATABRICKS_HTTP_PATH"),
    catalog="my_catalog",
    schema="my_schema",
    access_token=os.environ.get("DATABRICKS_ACCESS_TOKEN")
)

This fails during the _post_connect() call when trying to create the memtable volume.

Root Cause

The issue appears to be in the volume name generation logic called from:

ibis/ibis/backends/databricks/__init__.py

Line 418 in 9126733

self._post_connect(memtable_volume=memtable_volume)

The _post_connect(memtable_volume=memtable_volume) method attempts to create a volume, but the three-level namespace (catalog.schema.volume) is not properly constructed.

Environment

• ibis-framework version: 11.0.0
• Python version: 3.13
• Operating System: macOS
• Databricks Runtime: SQL Warehouse
• Unity Catalog: Enabled

Workaround

Currently working around this by overriding _post_connect():

from ibis.backends.databricks import Backend

class NoDatabricksMemtable(Backend):
    def _post_connect(self, memtable_volume=None):
        pass  # Skip memtable initialization

con = NoDatabricksMemtable()
con.do_connect(
    server_hostname=server_hostname,
    http_path=http_path,
    access_token=access_token,
    catalog=catalog,
    schema=schema
)

Additional Context

• Specifying temp_directory explicitly does not help
• Setting memtable_volume=None does not prevent the error
• The issue occurs even with a pre-created volume

This makes it difficult to use ibis-framework with Unity Catalog in production environments, especially in notebook environments like marimo.

What version of ibis are you using?

ibis-framework version: 11.0.0

What backend(s) are you using, if any?

Databricks SQL Warehouse

Relevant log output

Traceback (most recent call last):
  File "/tmp/marimo_8608/__marimo__cell_Hbol_.py", line 10, in <module>
    engine = ibis.databricks.connect(
        server_hostname=_server_hostname,
    ...<4 lines>...
        memtable_volume=None
    )
  File "/.venv/lib/python3.13/site-packages/ibis/__init__.py", line 110, in connect
    return backend.connect(*args, **kwargs)
  File "/.venv/lib/python3.13/site-packages/ibis/backends/__init__.py", line 1050, in connect
    new_backend.reconnect()
  File "/.venv/lib/python3.13/site-packages/ibis/backends/__init__.py", line 1065, in reconnect
    self.do_connect(*self._con_args, **self._con_kwargs)
  File "/.venv/lib/python3.13/site-packages/ibis/backends/databricks/__init__.py", line 418, in do_connect
    self._post_connect(memtable_volume=memtable_volume)
  File "/.venv/lib/python3.13/site-packages/ibis/backends/databricks/__init__.py", line 446, in _post_connect
    cur.execute(sql)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/telemetry/latency_logger.py", line 175, in wrapper
    result = func(self, *args, **kwargs)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/client.py", line 968, in execute
    self.active_result_set = self.backend.execute_command(...)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/backend/thrift_backend.py", line 1058, in execute_command
    execute_response, has_more_rows = self._handle_execute_response(...)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/backend/thrift_backend.py", line 1265, in _handle_execute_response
    final_operation_state = self._wait_until_command_done(...)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/backend/thrift_backend.py", line 944, in _wait_until_command_done
    self._check_command_not_in_error_or_closed_state(...)
  File "/.venv/lib/python3.13/site-packages/databricks/sql/backend/thrift_backend.py", line 635, in _check_command_not_in_error_or_closed_state
    raise ServerOperationError(...)
databricks.sql.exc.ServerOperationError: [UC_VOLUME_NOT_FOUND] Volume `my_catalogmy_schemausername-py=3131-pid=8608` does not exist. 
Please use 'SHOW VOLUMES' to list available volumes. SQLSTATE: 42704

Code of Conduct

• I agree to follow this project's Code of Conduct