Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD

[DilmurodMak]

Pull Request Overview

This PR updates and enhances the Databricks deployment process using Terraform and Asset Bundle Deployment via GitHub Actions. It simplifies deployment for multi environment deployment.

Key Highlights

• Folder Structure:
  • Sample Code is Organized under single_tech_samples/databricks/databricks_terraform
  • Includes directories for:
    • Infra: Terraform code for Azure resource deployment.
    • tests: Scripts and workflows for Databricks testing.
    • utils: Helper scripts like generate-databricks-workflows.sh.
    • workflows: Pre-generated Databricks workflows to run tests in Databricks workspace.
• CI/CD Pipelines:
  • Linting Pipeline: Validates Python notebooks and workflows.
  • Sandbox Deployment: Validates and deploys Databricks assets, executes test workflows.
  • Development Deployment: Deploys assets to the development environment following successful sandbox deployment and runs the same tests in development environment

Testing Steps

The Sample code covers the deployment from sandbox to development environment.

1. Create a new branch and submit a PR to main.
2. Verify the following:
   • Linting Pipeline runs and validates code.
   • Sandbox Deployment Pipeline validates and tests workflows during the PR.
3. Merge the PR to main.
4. Observe:
   • Sandbox assets are deployed, and test workflows are executed successfully.
   • Development deployment triggers upon successful sandbox completion.

[ydaponte]

@DilmurodMak - one of the validations is failing, can you take a look?

[DilmurodMak]

@DilmurodMak - one of the validations is failing, can you take a look?

@ydaponte , The pipeline are templates, it requires databricks workspaces exist and its urls are set in databricks.yml file. There for its failing. We can put it as template reference in the single solution doc instead of running in PRs if we do not want to trigger it

[ydaponte]

Leaving some comments that need to be addressed before we can merge into main. There are some best practices and alignment with the overall repo that will need to be done as for example the creation of a devcontainer for the sample. Thanks for the great work so far!

[ydaponte]

I can't run the container - the req.txt file is missing. Can you upload it so I can further test? Thanks!

[ydaponte]

Just leaving the comment as we discussed earlier today.

[ydaponte]

It seems that being on the account_unity_admin group is not working. Let's try to talk today. Tks!

[ydaponte]

Approved! Thanks for the great work!