BadRequestError when reading Databricks Unity Catalog

[bolinzzz]

Hi team,

I’m using Daft version 0.4.3 and trying to load a Databricks Unity Catalog table as follows:

import daft
from daft.unity_catalog import UnityCatalog

unity = UnityCatalog(
    endpoint = "https://{}.cloud.databricks.com",
    token="",
)

unity_table = unity.load_table("")

I encounter BadRequestError

BadRequestError: Error code: 400 - {'error_code': 'INVALID_PARAMETER_VALUE', 'message': "Table with id 1f5e5b5c-4133-4a08-ab99-0e69e3803e4d cannot be written from outside of Databricks Compute Environment due to its kind being TABLE_DELTA. Only 'TABLE_EXTERNAL' and 'TABLE_DELTA_EXTERNAL' table kinds can be written externally.", 'details': [{'@type': 'type.googleapis.com[/google.rpc.ErrorInfo](http://localhost:8888/google.rpc.ErrorInfo)', 'reason': 'EXTERNAL_WRITE_NOT_ALLOWED_FOR_TABLE', 'domain': 'unity-catalog.databricks.com', 'metadata': {'tableId': '1f5e5b5c-4133-4a08-ab99-0e69e3803e4d', 'securableKind': 'TABLE_DELTA'}}, {'@type': 'type.googleapis.com[/google.rpc.RequestInfo](http://localhost:8888/google.rpc.RequestInfo)', 'request_id': '1dc2eeee-bead-4109-a1da-3c9215911a9e', 'serving_data': ''}]}

It appears that Daft can only read external tables in Delta format and does not support reading tables stored in the default Databricks-managed storage for a metastore. Is this statement accurate?

Thanks!

[raunakab]

Hey @bolinzzz, looking into this currently.

[gatb27]

I am also running into the same issue even though on the Databricks documentation is reported that also MANAGED TABLES are supported: https://docs.databricks.com/aws/en/external-access/credential-vending

[colin-ho]

Yes, you cannot write to managed tables in all cases. As your managed table already exists. There is a write mode which you have to set to READ in the load table function, e.g. uc.load_table("table", operation="READ")