Operationalizing Data Quality validation for the enterprise

Two modules prior, we learned to run a Data Quality task to populate the Data Quality scores in Dataplex Catalog. In the last module, we learned to automate a Data Quality validation task on the Customer Master Data from Airflow on Cloud Composer. In this module, we will add to the existing Airflow DAG, steps to publish Data Quality scores as tags in the Data Quality pipeline from the prior module. This is representative of the pattern for operationalizing for production.

For Data Quality tagging, we will use a Spark application that we will run via Dataplex tasks.

End to End Architecture for Data Quality validation

Learning goals

We have learned how to author Dataplex DQ YAML, run from command line and automate with Airflow on Cloud Composer. Separately, we learned how to tag Data Quality scores to our Catalog entry. We also learned how to set up incident management for our DQ tasks. In this lab module, we bring it all together and show you how to operationalize Data Quality validation.

Prerequisites

Successful completion of prior modules

Duration

15 minutes or less

Documentation

Data Quality Task orchestration with Airflow

Lab flow

LAB

1. The Airflow DAG

This is the DAG we ran in the prior lab module.

In this lab module, we will run a DAG that includes additional steps - to push Data Quality scores to Data Catalog DQ tags for our Data Catalog entry for the customer_master table.

2. Review the DAG code

The DAG code is available in your cloud shell already and is called "customer_master_dq_automation_e2e.py". Study how the Data Quality score tagging is a plain Spark application. But still needs polling as Dataplex tasks are async.

cat ~/dataplex-quickstart-labs/00-resources/scripts/airflow/data-quality/customer_master_dq_automation_e2e.py

4. Upload the Airflow DAG for Data Quality

Lets upload the DAG to the Cloud Composer DAG bucket.

PROJECT_ID=`gcloud config list --format "value(core.project)" 2>/dev/null`
PROJECT_NBR=`gcloud projects describe $PROJECT_ID | grep projectNumber | cut -d':' -f2 |  tr -d "'" | xargs`
LOCATION="us-central1"

# DAG bucket to upload to
DAG_BUCKET=`gcloud composer environments describe oda-$PROJECT_NBR-cc2 --location $LOCATION | grep dagGcsPrefix: | cut -d' ' -f4`

# Upload the DAG
gsutil cp ~/dataplex-quickstart-labs/00-resources/scripts/airflow/data-quality/customer_master_dq_automation_e2e.py $DAG_BUCKET/data-quality/