Skip to content

docs: Missing databricks backend docs #10969

Open
@filipeo2-mck

Description

@filipeo2-mck

Please describe the issue

Hello!

First: thanks for this amazing project :)

I noted that databricks is now enabled as a new backend (#10223, it's also listed under Backend support matrix), but I see no documentation on how to make the connection properly.

Digging into the source code, I noted

  • the need for DATABRICKS_SERVER_HOSTNAME, DATABRICKS_HTTP_PATH and DATABRICKS_TOKEN env vars
  • the .connect() needs additional arguments:
    con = ibis.databricks.connect(
        catalog="main",
        schema="default",
    )
  • it relies on Unity Catalog Volumes. At some point of the connection, it tries to create a Volume which is available only in the premium tiers (the need for Unity Catalog). When using a non-premium, I got:
    ServerOperationError: [[UC_VOLUMES_NOT_ENABLED](https://docs.microsoft.com/azure/databricks/error-messages/error-classes#uc_volumes_not_enabled)] Support for Unity Catalog Volumes is not enabled on this instance. SQLSTATE: 56038
    == SQL ==
    CREATE VOLUME IF NOT EXISTS `root-py=31012-pid=2012` COMMENT 'Ibis memtable storage volume'

It would be awesome if those (and additional) minimal requirements were declared under the "Backends" section of the docs.

I've tested it using ibis-framework==10.2.0, Python 3.10.

Thank you.

Code of Conduct

  • I agree to follow this project's Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    docsDocumentation related issues or PRs

    Type

    No type

    Projects

    Status

    backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions