Merge branch 'release/1.3.0' into 497-enhance-databricks-catalog-connector-documentation

Author: kczimm

website/docs/components/catalogs/databricks.md

@@ -11,7 +11,7 @@ tags:
   - data-connectors
 ---
 
-Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html) or directly from [Delta Lake](https://delta.io/) tables.
+Connect to a [Databricks Unity Catalog](https://www.databricks.com/product/unity-catalog) as a catalog provider for federated SQL query using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html), directly from [Delta Lake](https://delta.io/) tables, or using the [SQL Statement Execution API](https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial).
 
 ## Configuration
 
@@ -22,7 +22,7 @@ catalogs:
     include:
       - '*.my_table_name' # include only the "my_table_name" tables
     params:
-      mode: delta_lake # or spark_connect
+      mode: delta_lake # or spark_connect or sql_warehouse
       databricks_endpoint: dbc-a12cd3e4-56f7.cloud.databricks.com
     dataset_params:
       # delta_lake S3 parameters
@@ -32,6 +32,8 @@ catalogs:
       databricks_aws_endpoint: s3.us-west-2.amazonaws.com
       # spark_connect parameters
       databricks_cluster_id: 1234-567890-abcde123
+      # sql_warehouse parameters
+      databricks_sql_warehouse_id: 2b4e24cff378fb24
 ```
 
 ## `from`
@@ -57,6 +59,14 @@ The following parameters are supported for configuring the connection to the Dat
 | `databricks_token` | The Databricks API token to authenticate with the Unity Catalog API. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:my_databricks_token}`. |
 | `databricks_use_ssl` | If true, use a TLS connection to connect to the Databricks endpoint. Default is `true`. |
 
+To locate the Databricks endpoint, do the following:
+
+1. Log in to your Databricks workspace.
+2. In the sidebar, click Compute.
+3. In the list of available clusters, click the target cluster's name.
+4. On the Configuration tab, expand Advanced options.
+5. Click the JDBC/ODBC tab.
+6. The endpoint is the Server Hostname.
 
 ## Authentication
 
@@ -105,10 +115,30 @@ The `dataset_params` field is used to configure the dataset-specific parameters
 |------------------------|------------|
 | `databricks_cluster_id` | The ID of the compute cluster in Databricks to use for the query. e.g. `1234-567890-abcde123`. |
 
+To locate the cluster ID, do the following:
+
+1. Log in to your Databricks workspace.
+2. In the sidebar, click Compute.
+3. In the list of available clusters, click the target cluster's name.
+4. On the Configuration tab, expand Advanced options.
+5. Click the JDBC/ODBC tab.
+6. The cluster ID is the prefix of the Server Hostname.
+
 ### Delta Lake object store parameters
 
 Configure the connection to the object store when using `mode: delta_lake`. Use the [secret replacement syntax](../secret-stores/index.md) to reference a secret, e.g. `${secrets:aws_access_key_id}`.
 
+### SQL Warehouse parameters
+
+- `databricks_sql_warehouse_id`: The ID of the SQL Warehouse in Databricks to use for the query. e.g. `2b4e24cff378fb24`.
+
+To locate your SQL Warehouse ID, do the following:
+
+1. Log in to your Databricks workspace.
+2. In the sidebar, click SQL -> SQL Warehouses.
+3. In the list of available warehouses, click the target warehouse's name.
+4. Next to the **Name** field, the ID follows the name in parentheses. For example: `My Serverless Warehouse (ID: 2b4e24cff378fb24)`
+
 #### AWS S3
 
 | Dataset Parameter Name | Definition |

website/docs/components/data-connectors/databricks.md

@@ -9,7 +9,7 @@ tags:
   - delta-lake
 ---
 
-Databricks as a connector for federated SQL query against Databricks using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html) or directly from [Delta Lake](https://delta.io/) tables.
+Databricks as a connector for federated SQL query against Databricks using [Spark Connect](https://www.databricks.com/blog/2022/07/07/introducing-spark-connect-the-power-of-apache-spark-everywhere.html), directly from [Delta Lake](https://delta.io/) tables, or using the [SQL Statement Execution API](https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial).
 
 ```yaml
 datasets:
@@ -62,6 +62,7 @@ Use the [secret replacement syntax](../secret-stores/index.md) to reference a se
 | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `mode`                     | The execution mode for querying against Databricks. The default is `spark_connect`. Possible values:<br /> <ul><li>`spark_connect`: Use Spark Connect to query against Databricks. Requires a Spark cluster to be available.</li><li>`delta_lake`: Query directly from Delta Tables. Requires the object store credentials to be provided.</li></ul> |
 | `databricks_endpoint`      | The endpoint of the Databricks instance. Required for both modes.                                                                                                                                                                                                                                                                                    |
+| `databricks_sql_warehouse_id`    | The ID of the SQL Warehouse in Databricks to use for the query. Only valid when `mode` is `sql_warehouse`.                                                                                                                                                                                                                                         |
 | `databricks_cluster_id`    | The ID of the compute cluster in Databricks to use for the query. Only valid when `mode` is `spark_connect`.                                                                                                                                                                                                                                         |
 | `databricks_use_ssl`       | If true, use a TLS connection to connect to the Databricks endpoint. Default is `true`.                                                                                                                                                                                                                                                              |
 | `client_timeout`           | Optional. Applicable only in `delta_lake` mode. Specifies timeout for object store operations. Default value is `30s` E.g. `client_timeout: 60s`                                                                                                                                                                                                     |
@@ -157,6 +158,18 @@ Configure the connection to the object store when using `mode: delta_lake`. Use
     databricks_token: ${secrets:my_token}
 ```
 
+### SQL Warehouse
+
+```yaml
+- from: databricks:spiceai.datasets.my_table # A reference to a table in the Databricks unity catalog
+  name: my_table
+  params:
+    mode: sql_warehouse
+    databricks_endpoint: dbc-a1b2345c-d6e7.cloud.databricks.com
+    databricks_sql_warehouse_id: 2b4e24cff378fb24
+    databricks_token: ${secrets:my_token}
+```
+
 ### Delta Lake (S3)
 
 ```yaml
@@ -259,4 +272,4 @@ Memory limitations can be mitigated by storing acceleration data on disk, which
 
 ## Cookbook
 
-- A cookbook recipe to configure Databricks as data connector in Spice under `delta_lake` mode. [Spice on Databricks (mode: delta_lake)](https://github.com/spiceai/cookbook/tree/trunk/databricks/delta_lake#readme)
+- A cookbook recipe to configure Databricks as a data connector in Spice. [Spice on Databricks](https://github.com/spiceai/cookbook/tree/trunk/databricks)