Load and save checks from a Delta table #339

ghanse · 2025-05-06T14:31:03Z

Changes

Added the following methods to the DQEngine class to save and load checks to a Delta table in a Databricks workspace:

save_checks_in_dataframe
load_checks_from_dataframe
save_checks_in_table
load_checks_from_table

Linked issues

Resolves #299

Tests

Added unit and integration tests.

manually tested
added unit tests
added integration tests

github-actions · 2025-05-06T14:40:07Z

✅ 149/149 passed, 1 flaky, 1 skipped, 1h13m36s total

Flaky tests:

🤪 test_profiler_workflow_e2e (5m16.636s)

_{Running from acceptance #724}

src/databricks/labs/dqx/engine.py

tests/integration/test_load_checks_from_table.py

mwojtyczka · 2025-05-08T22:48:40Z

src/databricks/labs/dqx/engine.py

+
+    @staticmethod
+    def _load_checks_from_table(
+        table_name: str, query: str | None = None, spark: SparkSession | None = None


why we need the query?

In case users want to filter checks on-load (e.g. where criticality <> "warning").

IMO it would be more useful to be able to load a set of checks for particular use case / run config. We already have the concept of run config. I would use run_config_name instead of query when saving and loading. By default use default run config.
I would not make it too flexible because it increases usage complexity.

I removed query. We can add run_config_name if needed.

Copilot

Pull Request Overview

This PR introduces the ability to persist data quality checks to and from Delta tables, enhances the core engine with DataFrame-based serialization/deserialization, and updates configuration and tests to support a checks_table.

Added save_checks_in_dataframe, load_checks_from_dataframe, save_checks_in_table, and load_checks_from_table in DQEngineCore/DQEngine.
Extended RunConfig with a checks_table setting and updated integration fixtures.
Bumped linter max-attributes to accommodate the new config field.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/unit/test_load_checks_from_dataframe.py	Unit tests for `load_checks_from_dataframe`, including warning.
tests/integration/test_load_checks_from_table.py	Integration tests for loading checks from a Delta table.
tests/integration/conftest.py	Updated `MockInstallationContext` to inject `checks_table`.
src/databricks/labs/dqx/engine.py	Implemented DataFrame/table save/load methods and warning logic.
src/databricks/labs/dqx/config.py	Added `checks_table` field to `RunConfig`.
pyproject.toml	Increased `max-attributes` limit to 16.

src/databricks/labs/dqx/engine.py

tests/unit/test_load_checks_from_dataframe.py

src/databricks/labs/dqx/engine.py

mwojtyczka · 2025-05-20T15:03:09Z

src/databricks/labs/dqx/engine.py

+
+    @staticmethod
+    def _load_checks_from_table(
+        table_name: str, query: str | None = None, spark: SparkSession | None = None


IMO it would be more useful to be able to load a set of checks for particular use case / run config. We already have the concept of run config. I would use run_config_name instead of query when saving and loading. By default use default run config.
I would not make it too flexible because it increases usage complexity.

tests/integration/test_load_checks_from_table.py

Copilot

Pull Request Overview

Adds Delta table persistence for data quality checks by introducing DataFrame conversion utilities and new DQEngine methods for saving/loading to tables.

Implemented build_dataframe_from_quality_rules and build_quality_rules_from_dataframe in DQEngineCore
Added save_checks_in_table and load_checks_from_table in both DQEngineCore and DQEngine
Updated configuration and tests to support table-based check persistence

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/databricks/labs/dqx/engine.py	Added build/save/load methods for DataFrame and Delta table persistence of checks
src/databricks/labs/dqx/config.py	Introduced `checks_table` field in `RunConfig`
tests/unit/test_build_rules.py	Added unit test for DataFrame-based quality rules build
tests/integration/test_load_checks_from_table.py	Added integration tests for loading checks from a Delta table
tests/integration/conftest.py	Adjusted `installation_ctx` fixture signature formatting
pyproject.toml	Increased `max-attributes` limit to accommodate new methods

Comments suppressed due to low confidence (1)

tests/unit/test_build_rules.py:446

Test covers basic DataFrame conversion but does not verify behavior when a filter column is present or when a large-DataFrame warning is emitted. Consider adding tests for those scenarios.

def test_build_quality_rules_from_dataframe(spark_local):

Copilot · 2025-05-21T15:16:05Z

src/databricks/labs/dqx/engine.py

+        * `filter` - Expression for filtering data quality checks
+        :return: list of data quality check specifications as a Python dictionary
+        """
+        num_check_rows = df.count()


Calling df.count() before df.collect() triggers two separate Spark jobs. Consider collecting rows once (e.g., rows = df.collect()) and using len(rows) to avoid the extra scan.

Suggested change

num_check_rows = df.count()

rows = df.collect()

num_check_rows = len(rows)

Copilot · 2025-05-21T15:16:06Z

src/databricks/labs/dqx/engine.py

+        * `name` - name that will be given to a resulting column. Autogenerated if not provided
+        * `criticality` (optional) - possible values are `error` (data going only into "bad" dataframe),
+        and `warn` (data is going into both dataframes)
+        * `check_function` - DQX check function used in the check


The docstring refers to a check_function column, but the actual DataFrame schema uses a check column. Update the description to match the implementation.

Suggested change

* `check_function` - DQX check function used in the check

* `check` - DQX check function used in the check

Copilot · 2025-05-21T15:16:06Z

src/databricks/labs/dqx/engine.py

+        if spark is None:
+            spark = SparkSession.builder.getOrCreate()
+        rules_df = spark.read.table(table_name)
+        return DQEngineCore.build_quality_rules_from_dataframe(rules_df)


[nitpick] This static method duplicates functionality in DQEngineCore. Consider consolidating with the core implementation to reduce code duplication.

Suggested change

if spark is None:

spark = SparkSession.builder.getOrCreate()

rules_df = spark.read.table(table_name)

return DQEngineCore.build_quality_rules_from_dataframe(rules_df)

"""

Load checks from a Delta table in the workspace.

:param table_name: Unity catalog or Hive metastore table name

:param spark: Optional SparkSession instance

:return: List of quality rules (checks)

"""

return DQEngineCore.load_checks_from_table(table_name, spark)

Load and save checks from a Delta table

a1c768e

ghanse requested a review from a team as a code owner May 6, 2025 14:31

ghanse requested review from tombonfert and removed request for a team May 6, 2025 14:31

ghanse had a problem deploying to tool May 6, 2025 14:31 — with GitHub Actions Failure

Use table_exists from the tables.exists response

2a8397e

ghanse had a problem deploying to tool May 6, 2025 15:45 — with GitHub Actions Failure

ghanse added 2 commits May 6, 2025 12:00

Create catalog and schema for tests

a9b8e42

Create catalog and schema for tests

1cc2a91

ghanse had a problem deploying to tool May 6, 2025 16:03 — with GitHub Actions Failure

mwojtyczka requested changes May 8, 2025

View reviewed changes

ghanse added 2 commits May 13, 2025 15:14

Updated and added tests

81a9af0

Cleaned up and added methods to load checks from a DataFrame

a4ae4bd

ghanse had a problem deploying to tool May 13, 2025 19:14 — with GitHub Actions Failure

ghanse had a problem deploying to tool May 13, 2025 19:14 — with GitHub Actions Error

Bump max-attributes for installation context

c85e575

ghanse had a problem deploying to tool May 13, 2025 19:23 — with GitHub Actions Failure

Updated tests

20f5e42

ghanse requested a review from mwojtyczka May 14, 2025 11:57

mwojtyczka requested a review from Copilot May 20, 2025 14:44

Copilot AI reviewed May 20, 2025

View reviewed changes

src/databricks/labs/dqx/engine.py Outdated Show resolved Hide resolved

src/databricks/labs/dqx/engine.py Outdated Show resolved Hide resolved

tests/unit/test_load_checks_from_dataframe.py Outdated Show resolved Hide resolved

mwojtyczka requested changes May 20, 2025

View reviewed changes

Update method names and docstrings; Remove query attribute

0e2cb04

ghanse had a problem deploying to tool May 20, 2025 17:27 — with GitHub Actions Failure

Fix tests

c842bf5

ghanse had a problem deploying to tool May 20, 2025 19:39 — with GitHub Actions Failure

Fix tests

0882b19

ghanse had a problem deploying to tool May 20, 2025 20:02 — with GitHub Actions Failure

Fix tests

3ca046f

ghanse had a problem deploying to tool May 20, 2025 20:15 — with GitHub Actions Failure

Fix tests

60a2b6e

ghanse had a problem deploying to tool May 20, 2025 20:38 — with GitHub Actions Failure

Fix tests

88bb6e3

ghanse had a problem deploying to tool May 20, 2025 20:56 — with GitHub Actions Failure

ghanse had a problem deploying to tool May 20, 2025 20:56 — with GitHub Actions Error

Return "filter" only if non-null

b3c9392

ghanse temporarily deployed to tool May 20, 2025 21:04 — with GitHub Actions Inactive

mwojtyczka requested a review from Copilot May 21, 2025 15:13

Merge branch 'main' into checks_from_table

3bcfc93

mwojtyczka temporarily deployed to tool May 21, 2025 15:13 — with GitHub Actions Inactive

mwojtyczka deployed to tool May 21, 2025 15:13 — with GitHub Actions Active

Copilot AI reviewed May 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Load and save checks from a Delta table #339

Load and save checks from a Delta table #339

ghanse commented May 6, 2025 •

edited

Loading

github-actions bot commented May 6, 2025 •

edited

Loading

mwojtyczka May 8, 2025

ghanse May 13, 2025

mwojtyczka May 20, 2025

ghanse May 20, 2025

Copilot AI left a comment

mwojtyczka May 20, 2025

Copilot AI left a comment

Copilot AI May 21, 2025

Copilot AI May 21, 2025

Copilot AI May 21, 2025

	num_check_rows = df.count()
	rows = df.collect()
	num_check_rows = len(rows)

	* `check_function` - DQX check function used in the check
	* `check` - DQX check function used in the check

-        if spark is None:
-            spark = SparkSession.builder.getOrCreate()
-        rules_df = spark.read.table(table_name)
-        return DQEngineCore.build_quality_rules_from_dataframe(rules_df)
+        """
+        Load checks from a Delta table in the workspace.
+        :param table_name: Unity catalog or Hive metastore table name
+        :param spark: Optional SparkSession instance
+        :return: List of quality rules (checks)
+        """
+        return DQEngineCore.load_checks_from_table(table_name, spark)

Load and save checks from a Delta table #339

Are you sure you want to change the base?

Load and save checks from a Delta table #339

Conversation

ghanse commented May 6, 2025 • edited Loading

Changes

Linked issues

Tests

github-actions bot commented May 6, 2025 • edited Loading

mwojtyczka May 8, 2025

Choose a reason for hiding this comment

ghanse May 13, 2025

Choose a reason for hiding this comment

mwojtyczka May 20, 2025

Choose a reason for hiding this comment

ghanse May 20, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

mwojtyczka May 20, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI May 21, 2025

Choose a reason for hiding this comment

Copilot AI May 21, 2025

Choose a reason for hiding this comment

Copilot AI May 21, 2025

Choose a reason for hiding this comment

ghanse commented May 6, 2025 •

edited

Loading

github-actions bot commented May 6, 2025 •

edited

Loading