Hands-on labs for governing data and AI with Unity Catalog, MLflow, and Databricks Lakehouse Monitoring
Unity Catalog | Access Control | Data Lineage | CI/CD | MLOps | Secrets Management
Learn to govern production data platforms and ML systems on Databricks. This course covers:
- Unity Catalog — Centralized governance with metastores, catalogs, schemas, and tables
- Access Control — GRANT/REVOKE permissions, row/column-level security, and least-privilege design
- Data Lineage — Automatic lineage capture and compliance querying via system tables
- CI/CD for Notebooks — GitHub Actions, multi-environment promotion (dev → staging → prod)
- MLOps — MLflow experiment tracking, model registration, and promotion through the model registry
- Lakehouse Monitoring — Data quality alerts, drift detection, and retraining triggers
- Secrets Management — Databricks secret scopes, Azure Key Vault, and AWS Secrets Manager integration
- Sign up for a Databricks workspace (Unity Catalog enabled by default on new workspaces)
- Clone this repository:
git clone https://github.com/alfredodeza/databricks-governance.git
- Upload example files from
examples/to your Databricks workspace - Run SQL examples in a SQL warehouse or attach Python notebooks to a cluster
- Follow the labs in
labs/for guided exercises
| Example | Files | Description |
|---|---|---|
| Unity Catalog | examples/unity/ |
Create catalogs, schemas, tables, and volumes |
| Access Control | examples/access-control/ |
GRANT/REVOKE and permission model design |
| Lineage | examples/lineage/ |
Query lineage system tables |
| Repos / Git | examples/repos/ |
GitHub Actions workflow for Databricks Repos |
| CI/CD for Notebooks | examples/cicd/ |
Multi-environment notebook deployment |
| MLOps | examples/mlops/ |
MLflow training, registration, and promotion |
| Lakehouse Monitoring | examples/monitoring/ |
Table monitors and alerting |
| ML Monitoring | examples/ml-monitoring/ |
Inference table drift detection |
| Secrets | examples/secrets/ |
Secret scopes and credential retrieval |
| Lab | Topic | Examples |
|---|---|---|
| Lab 1 | Unity Catalog Setup | unity/ |
| Lab 2 | Access Control | access-control/ |
| Lab 3 | Data Lineage | lineage/ |
| Lab 4 | CI/CD for Notebooks | repos/, cicd/ |
| Lab 5 | ML Model CI/CD | mlops/ |
| Lab 6 | Monitoring and Secrets | monitoring/, ml-monitoring/, secrets/ |
- Lesson 1.1 – Unity Catalog Overview — Metastores, catalogs, schemas, tables
- Lesson 1.4 – Access Control — GRANT, REVOKE, and permission design
- Lesson 1.6 – Data Lineage — System table queries and compliance reporting
- Lesson 2.1 – Databricks Repos — GitHub integration and branching strategies
- Lesson 2.3 – CI/CD for Notebooks — GitHub Actions and multi-environment promotion
- Lesson 2.6 – ML Model CI/CD — MLflow lifecycle and model registry promotion
- Lesson 3.1 – Lakehouse Monitoring — Table monitors and anomaly alerting
- Lesson 3.3 – ML Model Monitoring — Drift detection and retraining triggers
- Lesson 3.6 – Secrets Management — Secret scopes, Key Vault, and Secrets Manager
See the full course outline for detailed lesson breakdowns.
databricks-governance/
├── examples/
│ ├── unity/ # Unity Catalog: catalogs, schemas, tables, volumes
│ ├── access-control/ # GRANT/REVOKE and permission model design
│ ├── lineage/ # Lineage system table queries
│ ├── repos/ # GitHub Actions for Databricks Repos
│ ├── cicd/ # CI/CD pipelines for notebook promotion
│ ├── mlops/ # MLflow training, registration, and promotion
│ ├── monitoring/ # Lakehouse Monitoring table monitors
│ ├── ml-monitoring/ # Inference table drift detection
│ └── secrets/ # Secret scopes and credential handling
├── labs/ # Hands-on lab instructions
├── docs/ # Capstone project and supplementary docs
├── data/ # Sample datasets for demonstrations
└── assets/ # Course banner and graphics
- Unity Catalog Documentation
- MLflow Model Registry
- Databricks Lakehouse Monitoring
- Databricks Secret Management
- GitHub Actions for Databricks
See CONTRIBUTING.md for guidelines.
- Fork the repository
- Create a feature branch
- Submit a pull request
Apache License 2.0 — see LICENSE for details.
Made with care by Pragmatic AI Labs