Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

DilmurodMak · 2024-12-02T19:21:24Z

Pull Request Overview

This PR updates and enhances the Databricks deployment process using Terraform and Asset Bundle Deployment via GitHub Actions. It simplifies deployment for multi environment deployment.

Key Highlights

Folder Structure:
- Sample Code is Organized under single_tech_samples/databricks/databricks_terraform
- Includes directories for:
  - Infra: Terraform code for Azure resource deployment.
  - tests: Scripts and workflows for Databricks testing.
  - utils: Helper scripts like generate-databricks-workflows.sh.
  - workflows: Pre-generated Databricks workflows to run tests in Databricks workspace.
CI/CD Pipelines:
- Linting Pipeline: Validates Python notebooks and workflows.
- Sandbox Deployment: Validates and deploys Databricks assets, executes test workflows.
- Development Deployment: Deploys assets to the development environment following successful sandbox deployment and runs the same tests in development environment

Testing Steps

The Sample code covers the deployment from sandbox to development environment.

Create a new branch and submit a PR to main.
Verify the following:
- Linting Pipeline runs and validates code.
- Sandbox Deployment Pipeline validates and tests workflows during the PR.
Merge the PR to main.
Observe:
- Sandbox assets are deployed, and test workflows are executed successfully.
- Development deployment triggers upon successful sandbox completion.

ydaponte · 2024-12-04T09:41:15Z

@DilmurodMak - one of the validations is failing, can you take a look?

DilmurodMak · 2024-12-04T20:20:57Z

@DilmurodMak - one of the validations is failing, can you take a look?

@ydaponte , The pipeline are templates, it requires databricks workspaces exist and its urls are set in databricks.yml file. There for its failing. We can put it as template reference in the single solution doc instead of running in PRs if we do not want to trigger it

…ying resources

ydaponte

Leaving some comments that need to be addressed before we can merge into main. There are some best practices and alignment with the overall repo that will need to be done as for example the creation of a devcontainer for the sample. Thanks for the great work so far!

single_tech_samples/databricks/databricks_terraform/README.md

ydaponte · 2024-12-16T11:14:00Z

single_tech_samples/databricks/databricks_terraform/Infra/README.md

+
+### Pre-requisites
+- Clone the repository
+- Install Terraform CLI if not installed already [Terraform Installation](https://learn.hashicorp.com/tutorials/terraform/install-cli)


You can add this pre-requirements to be installed when launching the devcontainer

single_tech_samples/databricks/databricks_terraform/Infra/README.md

ydaponte · 2024-12-16T11:20:28Z

...mples/databricks/databricks_terraform/images/databricks-asset-bundle-deploymeny-pipeline.png

Instead of the png, can you please commit the drawio.svg version file of this diagram? We are starting to create a standard in the repo for that.

These was originally images, I will try to recreate in draw.io if necessary

ydaponte · 2024-12-16T11:21:00Z

single_tech_samples/databricks/databricks_terraform/images/architecture.png

Is there an editable version of this diagram that can be commited instead?

Co-authored-by: Yennifer Santos <[email protected]>

ydaponte

I can't run the container - the req.txt file is missing. Can you upload it so I can further test? Thanks!

ydaponte · 2025-01-09T11:19:32Z

single_tech_samples/databricks/databricks_terraform/.devcontainer/Dockerfile

+  && apt clean
+
+# Copy and install dev dependencies
+COPY requirements-dev.txt /tmp/requirements-dev.txt


The requirements-dev.txt file is missing. The devcontainer build is failing with this error: ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref bfc31110-fef0-4a1d-8108-d3bf8d65de30::x6ufq090g3b3ngzwutfeemidk: "/requ
irements-dev.txt": not found

@ydaponte, I see it was missing. I just added.

Co-authored-by: Yennifer Santos <[email protected]>

ydaponte

Just leaving the comment as we discussed earlier today.

ydaponte · 2025-01-09T19:31:54Z

single_tech_samples/databricks/databricks_terraform/Infra/README.md

+
+**NOTE** - *When **`adb-workspace`** module runs it creates databricks workspace, and by default it creates a metastore in the same region. Databricks allows only **ONE METASTORE** per region. **`metastore-and-users`** module deploys new metastore with our required configurations, but we have to delete existing metastore prior running the module*
+
+**NOTE** - *During script execution you will receive `Error: cannot create metastore: This account with id <Account_ID> has reached the limit for metastores in region <Region>` * error. This is because we have reached the limit of metastores in the region. To fix this, we need to delete existing metastore and re-run the script.*


Can we change the script to check if the metastore already exists? If it doesn't exist we create it but if exists we should reuse it. In the environment where I'm testing we do have already a metastore and I can't delete it. I can point to a different region to workaround the problem, but realistically if someone wants to try chances are that the metastore already exists most of the time.

DilmurodMak added 11 commits November 19, 2024 15:35

linting pipeline

22247d7

Infra Deployment

931373d

terraform samples azure databricks

70dd64e

asset bundles terraform

820b75d

terraform samples databricks

1513ade

terraform code

0c19567

asset bundle linting code

b8fdf45

linting pipeline

3754af7

databricks asset bundle setup

254ff18

sandbox and dev deployment pipelines

56824a9

databricks asset bundles sample code

fa4e787

DilmurodMak requested review from hemaverma and ydaponte December 2, 2024 19:21

DilmurodMak had a problem deploying to sandbox December 2, 2024 19:21 — with GitHub Actions Failure

DilmurodMak added 2 commits December 2, 2024 15:04

yaml lint

080ca8e

doc lint

ed1e96a

DilmurodMak had a problem deploying to sandbox December 2, 2024 20:06 — with GitHub Actions Failure

DilmurodMak had a problem deploying to sandbox December 2, 2024 20:39 — with GitHub Actions Failure

yaml indentation issue

c7f4208

ydaponte assigned DilmurodMak Dec 4, 2024

ydaponte added the single-tech: azure-databricks Related to Azure Databricks single-tech sample label Dec 4, 2024

ydaponte added this to the E2E - parking_sensor_databricks - V1 milestone Dec 4, 2024

update varaible instruction

3c8f9e4

DilmurodMak had a problem deploying to sandbox December 6, 2024 14:20 — with GitHub Actions Failure

wait 120 sec metastore creation

47eb1bd

DilmurodMak had a problem deploying to sandbox December 11, 2024 21:02 — with GitHub Actions Failure

metastore name

d874dd6

DilmurodMak had a problem deploying to sandbox December 11, 2024 21:31 — with GitHub Actions Failure

updated using single shell script for deploying resources and destoro…

9417ead

…ying resources

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:27 — with GitHub Actions Failure

shell script cleanup

e32e68d

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:32 — with GitHub Actions Failure

doc update

50f2bcf

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:43 — with GitHub Actions Failure

ydaponte requested changes Dec 16, 2024

View reviewed changes

Update single_tech_samples/databricks/databricks_terraform/README.md

36cb922

Co-authored-by: Yennifer Santos <[email protected]>

DilmurodMak had a problem deploying to sandbox December 16, 2024 14:48 — with GitHub Actions Failure

Update single_tech_samples/databricks/databricks_terraform/README.md

82da343

Co-authored-by: Yennifer Santos <[email protected]>

DilmurodMak had a problem deploying to sandbox December 16, 2024 14:48 — with GitHub Actions Failure

update doc and add devcontainer

309e849

DilmurodMak requested a review from ydaponte December 16, 2024 18:22

DilmurodMak added 3 commits December 16, 2024 15:56

adding doc update

784b520

add visio link to the doc for images

e30e28b

merge main resolve conflcits

7a2695b

ydaponte requested changes Jan 9, 2025

View reviewed changes

DilmurodMak and others added 3 commits January 9, 2025 09:03

requirements-dev.txt file

b57e5d5

Update single_tech_samples/databricks/databricks_terraform/README.md

9c2a58e

Co-authored-by: Yennifer Santos <[email protected]>

remove non exisiting hyperlink

902ad4e

ydaponte requested changes Jan 13, 2025

View reviewed changes

DilmurodMak added 2 commits January 14, 2025 16:14

use existing metastore

d54081e

use existing metastore

31f9a88

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

DilmurodMak commented Dec 2, 2024

ydaponte commented Dec 4, 2024

DilmurodMak commented Dec 4, 2024 •

edited

Loading

ydaponte left a comment

ydaponte Dec 16, 2024

ydaponte Dec 16, 2024

DilmurodMak Dec 16, 2024

ydaponte Dec 16, 2024

ydaponte left a comment

ydaponte Jan 9, 2025

DilmurodMak Jan 9, 2025

ydaponte left a comment

ydaponte Jan 9, 2025


		NOTE - When `adb-workspace`* module runs it creates databricks workspace, and by default it creates a metastore in the same region. Databricks allows only ONE METASTORE per region. `metastore-and-users` module deploys new metastore with our required configurations, but we have to delete existing metastore prior running the module*

		NOTE - During script execution you will receive `Error: cannot create metastore: This account with id <Account_ID> has reached the limit for metastores in region <Region>` error. This is because we have reached the limit of metastores in the region. To fix this, we need to delete existing metastore and re-run the script.*

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Are you sure you want to change the base?

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Conversation

DilmurodMak commented Dec 2, 2024

Pull Request Overview

Key Highlights

Testing Steps

ydaponte commented Dec 4, 2024

DilmurodMak commented Dec 4, 2024 • edited Loading

ydaponte left a comment

Choose a reason for hiding this comment

ydaponte Dec 16, 2024

Choose a reason for hiding this comment

ydaponte Dec 16, 2024

Choose a reason for hiding this comment

DilmurodMak Dec 16, 2024

Choose a reason for hiding this comment

ydaponte Dec 16, 2024

Choose a reason for hiding this comment

ydaponte left a comment

Choose a reason for hiding this comment

ydaponte Jan 9, 2025

Choose a reason for hiding this comment

DilmurodMak Jan 9, 2025

Choose a reason for hiding this comment

ydaponte left a comment

Choose a reason for hiding this comment

ydaponte Jan 9, 2025

Choose a reason for hiding this comment

DilmurodMak commented Dec 4, 2024 •

edited

Loading