adb-overwatch-ws-to-monitor

adb-overwatch-ws-to-monitor

This module deploys the required resources for a given Databricks workspace to be monitored by Overwatch :

• Databricks PAT that will be used by Overwatch jobs
• Eventhub topic with its authorization rule
• Diagnostics settings
• AKV secrets to store the Databricks PAT, and the Eventhub primary connection string created above
• AKV-backed Databricks secret scope
• Container for the cluster logs in the existing log storage account
• Databricks mount to the container created above
• CSV file with all required parameters using Overwatch deployment template

Note
For more details on the column description, please refer to Overwatch Deployment Configuration

Requirements

No requirements.

Providers

Name	Version
azurerm	n/a
databricks	n/a
null	n/a
template	n/a

Modules

No modules.

Resources

Name	Type
azurerm_eventhub.eh	resource
azurerm_eventhub_authorization_rule.eh-ar	resource
azurerm_key_vault_secret.adb-pat	resource
azurerm_key_vault_secret.eh-conn-string	resource
azurerm_monitor_diagnostic_setting.dgs-ws	resource
azurerm_storage_data_lake_gen2_filesystem.cluster-logs-fs	resource
databricks_mount.cluster-logs-mount-ws	resource
databricks_secret_scope.overwatch-akv	resource
databricks_token.pat-ws	resource
null_resource.local	resource
azurerm_databricks_workspace.adb-ws	data source
azurerm_eventhub_namespace_authorization_rule.ehn-ar	data source
azurerm_key_vault.existing-kv	data source
azurerm_key_vault_secret.spn-key	data source
azurerm_monitor_diagnostic_categories.dgs-cat	data source
azurerm_resource_group.rg	data source
azurerm_storage_account.logs-sa	data source
template_cloudinit_config.local	data source
template_file.ow-deployment-config	data source

Inputs

Name	Description	Type	Default	Required
active	Whether or not the workspace should be validated / deployed	bool	n/a	yes
adb_ws_name	The name of an existing Databricks workspace that Overwatch will monitor	string	n/a	yes
akv_name	Azure Key-Vault name	string	n/a	yes
api_waiting_time	API Tunable - Overwatch makes async api calls in parallel, api_waiting_time signifies the max wait time in case of no response received from the api call	string	n/a	yes
auditlog_prefix_source_path	Location of auditlog (AWS/GCP Only)	string	n/a	yes
automated_dbu_price	Contract price for automated DBUs	number	n/a	yes
databricks_secret_scope_name	Databricks secret scope name (backed by Azure Key-Vault)	string	n/a	yes
ehn_auth_rule_name	Eventhub namespace authorization rule name	string	n/a	yes
ehn_name	Eventhub namespace name	string	n/a	yes
enable_unsafe_SSL	API Tunable - Enables unsafe SSL	string	n/a	yes
error_batch_size	API Tunable - Indicates the size of the error writer buffer containing API call errors	string	n/a	yes
etl_storage_prefix	Overwatch ETL storage prefix, which represents a mount point to the ETL storage account	string	n/a	yes
excluded_scopes	Scopes that should not be excluded from the pipelines	string	n/a	yes
interactive_dbu_price	Contract price for interactive DBUs	number	n/a	yes
jobs_light_dbu_price	Contract price for interactive DBUs	number	n/a	yes
logs_sa_name	Logs storage account name	string	n/a	yes
max_days	This is the max incremental days that will be loaded. Usually only relevant for historical loading and rebuilds	number	n/a	yes
overwatch_spn_app_id	Azure SPN used to create Databricks mounts	string	n/a	yes
proxy_host	Proxy url for the workspace	string	n/a	yes
proxy_password_key	Key which contains proxy password	string	n/a	yes
proxy_password_scope	Scope which contains the proxy password key	string	n/a	yes
proxy_port	Proxy port for the workspace	string	n/a	yes
proxy_user_name	Proxy user name for the workspace	string	n/a	yes
random_string	Random string used as a suffix for the resources names	string	n/a	yes
rg_name	Resource group name	string	n/a	yes
sql_compute_dbu_price	Contract price for DBSQL DBUs	number	n/a	yes
success_batch_size	API Tunable - Indicates the size of the buffer on filling of which the result will be written to a temp location. This is used to tune performance in certain circumstance	string	n/a	yes
tenant_id	Azure tenant ID	string	n/a	yes
thread_pool_size	API Tunable - Max number of API calls Overwatch is allowed to make in parallel	string	n/a	yes

Outputs

No outputs.