Skip to content

Commit 7439dc2

Browse files
committed
add docs to databricks catalog
1 parent 64916c4 commit 7439dc2

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

website/docs/components/catalogs/databricks.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ tags:
1111
- data-connectors
1212
---
1313

14-
Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html) or directly from [Delta Lake](https://delta.io/) tables.
14+
Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html), directly from [Delta Lake](https://delta.io/) tables, or using the [SQL Statement Execution API](https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial).
1515

1616
## Configuration
1717

@@ -22,7 +22,7 @@ catalogs:
2222
include:
2323
- '*.my_table_name' # include only the "my_table_name" tables
2424
params:
25-
mode: delta_lake # or spark_connect
25+
mode: delta_lake # or spark_connect or sql_warehouse
2626
databricks_endpoint: dbc-a12cd3e4-56f7.cloud.databricks.com
2727
dataset_params:
2828
# delta_lake S3 parameters
@@ -32,6 +32,8 @@ catalogs:
3232
databricks_aws_endpoint: s3.us-west-2.amazonaws.com
3333
# spark_connect parameters
3434
databricks_cluster_id: 1234-567890-abcde123
35+
# sql_warehouse parameters
36+
databricks_sql_warehouse_id: 1234-567890-abcde123
3537
```
3638
3739
## `from`
@@ -53,6 +55,7 @@ The `params` field is used to configure the connection to the Databricks Unity C
5355
- `mode`: The execution mode for querying against Databricks. The default is `spark_connect`. Possible values:
5456
- `spark_connect`: Use Spark Connect to query against Databricks. Requires a Spark cluster to be available.
5557
- `delta_lake`: Query directly from Delta Tables. Requires the object store credentials to be provided.
58+
- `sql_warehouse`: Use SQL Statement Execution API to query against a Databricks SQL Warehouse.
5659
- `databricks_endpoint`: The Databricks workspace endpoint, e.g. `dbc-a12cd3e4-56f7.cloud.databricks.com`.
5760
- `databricks_token`: The Databricks API token to authenticate with the Unity Catalog API. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:my_databricks_token}`.
5861
- `databricks_use_ssl`: If true, use a TLS connection to connect to the Databricks endpoint. Default is `true`.
@@ -106,6 +109,10 @@ The `dataset_params` field is used to configure the dataset-specific parameters
106109

107110
Configure the connection to the object store when using `mode: delta_lake`. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:aws_access_key_id}`.
108111

112+
### SQL Warehouse parameters
113+
114+
- `databricks_sql_warehouse_id`: The ID of the SQL Warehouse in Databricks to use for the query. e.g. `1234-567890-abcde123`.
115+
109116
#### AWS S3
110117

111118
- `databricks_aws_region`: The AWS region for the S3 object store. E.g. `us-west-2`.

0 commit comments

Comments
 (0)