Skip to content

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jan 27, 2025

Conversation

DilmurodMak
Copy link
Contributor

Pull Request Overview

This PR updates and enhances the Databricks deployment process using Terraform and Asset Bundle Deployment via GitHub Actions. It simplifies deployment for multi environment deployment.

Key Highlights

  • Folder Structure:
    • Sample Code is Organized under single_tech_samples/databricks/databricks_terraform
    • Includes directories for:
      • Infra: Terraform code for Azure resource deployment.
      • tests: Scripts and workflows for Databricks testing.
      • utils: Helper scripts like generate-databricks-workflows.sh.
      • workflows: Pre-generated Databricks workflows to run tests in Databricks workspace.
  • CI/CD Pipelines:
    • Linting Pipeline: Validates Python notebooks and workflows.
    • Sandbox Deployment: Validates and deploys Databricks assets, executes test workflows.
    • Development Deployment: Deploys assets to the development environment following successful sandbox deployment and runs the same tests in development environment

Testing Steps

The Sample code covers the deployment from sandbox to development environment.

  1. Create a new branch and submit a PR to main.
  2. Verify the following:
    • Linting Pipeline runs and validates code.
    • Sandbox Deployment Pipeline validates and tests workflows during the PR.
  3. Merge the PR to main.
  4. Observe:
    • Sandbox assets are deployed, and test workflows are executed successfully.
    • Development deployment triggers upon successful sandbox completion.

@ydaponte
Copy link
Collaborator

ydaponte commented Dec 4, 2024

@DilmurodMak - one of the validations is failing, can you take a look?

@ydaponte ydaponte added the single-tech: azure-databricks Related to Azure Databricks single-tech sample label Dec 4, 2024
@DilmurodMak
Copy link
Contributor Author

DilmurodMak commented Dec 4, 2024

@DilmurodMak - one of the validations is failing, can you take a look?

@ydaponte , The pipeline are templates, it requires databricks workspaces exist and its urls are set in databricks.yml file. There for its failing. We can put it as template reference in the single solution doc instead of running in PRs if we do not want to trigger it

Copy link
Collaborator

@ydaponte ydaponte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving some comments that need to be addressed before we can merge into main. There are some best practices and alignment with the overall repo that will need to be done as for example the creation of a devcontainer for the sample. Thanks for the great work so far!

@DilmurodMak DilmurodMak requested a review from ydaponte December 16, 2024 18:22
Copy link
Collaborator

@ydaponte ydaponte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't run the container - the req.txt file is missing. Can you upload it so I can further test? Thanks!

Copy link
Collaborator

@ydaponte ydaponte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just leaving the comment as we discussed earlier today.

Copy link
Collaborator

@ydaponte ydaponte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that being on the account_unity_admin group is not working. Let's try to talk today. Tks!

Copy link
Collaborator

@ydaponte ydaponte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved! Thanks for the great work!

@DilmurodMak DilmurodMak merged commit 46fbea4 into main Jan 27, 2025
3 checks passed
@ydaponte ydaponte deleted the dilmurod-Mak/asset-bundles branch January 28, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
single-tech: azure-databricks Related to Azure Databricks single-tech sample
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants