You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: website/docs/components/catalogs/databricks.md
+9-2Lines changed: 9 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,7 +11,7 @@ tags:
11
11
- data-connectors
12
12
---
13
13
14
-
Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html) or directly from [Delta Lake](https://delta.io/) tables.
14
+
Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html), directly from [Delta Lake](https://delta.io/) tables, or using the [SQL Statement Execution API](https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial).
15
15
16
16
## Configuration
17
17
@@ -22,7 +22,7 @@ catalogs:
22
22
include:
23
23
- '*.my_table_name'# include only the "my_table_name" tables
24
24
params:
25
-
mode: delta_lake # or spark_connect
25
+
mode: delta_lake # or spark_connect or sql_warehouse
@@ -53,6 +55,7 @@ The `params` field is used to configure the connection to the Databricks Unity C
53
55
- `mode`: The execution mode for querying against Databricks. The default is `spark_connect`. Possible values:
54
56
- `spark_connect`: Use Spark Connect to query against Databricks. Requires a Spark cluster to be available.
55
57
- `delta_lake`: Query directly from Delta Tables. Requires the object store credentials to be provided.
58
+
- `sql_warehouse`: Use SQL Statement Execution API to query against a Databricks SQL Warehouse.
56
59
- `databricks_endpoint`: The Databricks workspace endpoint, e.g. `dbc-a12cd3e4-56f7.cloud.databricks.com`.
57
60
- `databricks_token`: The Databricks API token to authenticate with the Unity Catalog API. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:my_databricks_token}`.
58
61
- `databricks_use_ssl`: If true, use a TLS connection to connect to the Databricks endpoint. Default is `true`.
@@ -106,6 +109,10 @@ The `dataset_params` field is used to configure the dataset-specific parameters
106
109
107
110
Configure the connection to the object store when using `mode: delta_lake`. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:aws_access_key_id}`.
108
111
112
+
### SQL Warehouse parameters
113
+
114
+
- `databricks_sql_warehouse_id`: The ID of the SQL Warehouse in Databricks to use for the query. e.g. `1234-567890-abcde123`.
115
+
109
116
#### AWS S3
110
117
111
118
- `databricks_aws_region`: The AWS region for the S3 object store. E.g. `us-west-2`.
0 commit comments