Release v0.10.0 (#920)

* Added Data Quality Summary Metrics
([#553](#553)). The data
quality engine has been enhanced with the ability to track and manage
summary metrics for data quality validation, leveraging Spark's
Observation feature. A new `DQMetricsObserver` class has been introduced
to manage Spark observations and track summary metrics on datasets
checked with the engine. The `DQEngine` class has been updated to
optionally return the Spark observation associated with a given run,
allowing users to access and save summary metrics. The engine now
supports also writing summary metrics to a table using the
`metrics_config` parameter, and a new `save_summary_metrics` method has
been added to save data quality summary metrics to a table.
Additionally, the engine has been updated to include a unique `run_id`
field in the detailed per-row quality results, enabling
cross-referencing with summary metrics. The changes also include updates
to the configuration file to support the storage of summary metrics.
Overall, these enhancements provide a more comprehensive and flexible
data quality checking capability, allowing users to track and analyze
data quality issues more effectively.
* LLM assisted rules generation
([#577](#577)). This release
introduces a significant enhancement to the data quality rules
generation process with the integration of AI-assisted rules generation
using large language models (LLMs). The `DQGenerator` class now includes
a `generate_dq_rules_ai_assisted` method, which takes user input in
natural language and optionally a schema from an input table to generate
data quality rules. These rules are then validated for correctness. The
AI-assisted rules generation feature supports both programmatic and
no-code approaches. Additionally, the feature enables the use of
different LLM models and gives the possibility to use custom check
functions. The release also includes various updates to the
documentation, configuration files, and testing framework to support the
new AI-assisted rules generation feature, ensuring a more streamlined
and efficient process for defining and applying data quality rules.
* Added Lakebase checks storage backend
([#550](#550)). A Lakebase
checks storage backend was added, allowing users to store and manage
their data quality rules in a centralized lakabase table, in addition to
the existing Delta table storage. The `checks_location` resolution has
been updated to accommodate Lakebase, supporting both table and file
storage, with flexible formatting options, including
"catalog.schema.table" and "database.schema.table". The Lakebase checks
storage backend is configurable through the
`LakebaseChecksStorageConfig` class, which includes fields for instance
name, user, location, port, run configuration name, and write mode. This
update provides users with more flexibility in storing and loading
quality checks, ensuring that checks are saved correctly regardless of
the specified location format.
* Added runtime validation of sql expressions
([#625](#625)). The data
quality check functionality has been enhanced with runtime validation of
SQL expressions, ensuring that specified fields can be resolved in the
input DataFrame and that SQL expressions are valid before evaluation. If
an SQL expression is invalid, the check evaluation is skipped and the
results include a check failure with a descriptive message.
Additionally, the configuration validation for Unity Catalog volume file
paths has been improved to enforce a specific format, preventing invalid
configurations and providing more informative error messages.
* Fixed docs ([#598](#598)).
The documentation build process has undergone significant improvements
to enhance efficiency and maintainability.
* Improved Config Serialization
([#676](#676)). Several
updates have been made to improve the functionality, consistency, and
maintainability of the codebase. The configuration loading functionality
has been refactored to utilize the `ConfigSerializer` class, which
handles the serialization and deserialization of workspace and run
configurations.
* Restore use of `hatch-fancy-pypi-readme` to fix images in PyPi
([#601](#601)). The image
source path for the logo in the README has been modified to correctly
display the logo image when rendered, particularly on PyPi.
* Skip check evaluation if columns or filter cannot be resolved in the
input DataFrame
([#609](#609)). DQX now skip
check evaluation if columns or filters are incorrect allowing other
checks to proceed even if one rule fails. The DQX engine validates
specified column, columns and filter fields against the input DataFrame
before applying checks, skipping evaluation and providing informative
error messages if any fields are invalid.
* Updated user guide docs
([#607](#607)). The
documentation for quality checking and integration options has been
updated to provide accurate and detailed information on supported types
and approaches. Quality checking can be performed in-transit
(pre-commit), validating data on the fly during processing, or at-rest,
checking existing data stored in tables.
* Improved build process
([#618](#618)). The hatch
version has been updated to 1.15.0 to avoid compatibility issues with
click version 8.3 and later, which introduced a bug affecting hatch.
Additionally, the project's dependencies have been updated, including
bumping the `databricks-labs-pytester` version from 0.7.2 to 0.7.4, and
code refactoring has been done to use a single Lakebase instance for all
integration tests, with retry logic added to handle cases where the
workspace quota limit for the number of Lakebase instances is exceeded,
enhancing the testing infrastructure and improving test reliability.
Furthermore, documentation updates have been made to clarify the
application of quality checks to data using DQX. These changes aim to
improve the efficiency, reliability, and clarity of the project's
testing and documentation infrastructure.

Author: mwojtyczka

CHANGELOG.md

@@ -1,5 +1,18 @@
 # Version changelog
 
+## 0.10.0
+
+* Added Data Quality Summary Metrics ([#553](https://github.com/databrickslabs/dqx/issues/553)). The data quality engine has been enhanced with the ability to track and manage summary metrics for data quality validation, leveraging Spark's Observation feature. A new `DQMetricsObserver` class has been introduced to manage Spark observations and track summary metrics on datasets checked with the engine. The `DQEngine` class has been updated to optionally return the Spark observation associated with a given run, allowing users to access and save summary metrics. The engine now supports also writing summary metrics to a table using the `metrics_config` parameter, and a new `save_summary_metrics` method has been added to save data quality summary metrics to a table. Additionally, the engine has been updated to include a unique `run_id` field in the detailed per-row quality results, enabling cross-referencing with summary metrics. The changes also include updates to the configuration file to support the storage of summary metrics. Overall, these enhancements provide a more comprehensive and flexible data quality checking capability, allowing users to track and analyze data quality issues more effectively.
+* LLM assisted rules generation ([#577](https://github.com/databrickslabs/dqx/issues/577)). This release introduces a significant enhancement to the data quality rules generation process with the integration of AI-assisted rules generation using large language models (LLMs). The `DQGenerator` class now includes a `generate_dq_rules_ai_assisted` method, which takes user input in natural language and optionally a schema from an input table to generate data quality rules. These rules are then validated for correctness. The AI-assisted rules generation feature supports both programmatic and no-code approaches. Additionally, the feature enables the use of different LLM models and gives the possibility to use custom check functions. The release also includes various updates to the documentation, configuration files, and testing framework to support the new AI-assisted rules generation feature, ensuring a more streamlined and efficient process for defining and applying data quality rules.
+* Added Lakebase checks storage backend ([#550](https://github.com/databrickslabs/dqx/issues/550)). A Lakebase checks storage backend was added, allowing users to store and manage their data quality rules in a centralized lakabase table, in addition to the existing Delta table storage. The `checks_location` resolution has been updated to accommodate Lakebase, supporting both table and file storage, with flexible formatting options, including "catalog.schema.table" and "database.schema.table". The Lakebase checks storage backend is configurable through the `LakebaseChecksStorageConfig` class, which includes fields for instance name, user, location, port, run configuration name, and write mode. This update provides users with more flexibility in storing and loading quality checks, ensuring that checks are saved correctly regardless of the specified location format. 
+* Added runtime validation of sql expressions ([#625](https://github.com/databrickslabs/dqx/issues/625)). The data quality check functionality has been enhanced with runtime validation of SQL expressions, ensuring that specified fields can be resolved in the input DataFrame and that SQL expressions are valid before evaluation. If an SQL expression is invalid, the check evaluation is skipped and the results include a check failure with a descriptive message. Additionally, the configuration validation for Unity Catalog volume file paths has been improved to enforce a specific format, preventing invalid configurations and providing more informative error messages.
+* Fixed docs ([#598](https://github.com/databrickslabs/dqx/issues/598)). The documentation build process has undergone significant improvements to enhance efficiency and maintainability. 
+* Improved Config Serialization ([#676](https://github.com/databrickslabs/dqx/issues/676)). Several updates have been made to improve the functionality, consistency, and maintainability of the codebase. The configuration loading functionality has been refactored to utilize the `ConfigSerializer` class, which handles the serialization and deserialization of workspace and run configurations. 
+* Restore use of `hatch-fancy-pypi-readme` to fix images in PyPi ([#601](https://github.com/databrickslabs/dqx/issues/601)). The image source path for the logo in the README has been modified to correctly display the logo image when rendered, particularly on PyPi.
+* Skip check evaluation if columns or filter cannot be resolved in the input DataFrame ([#609](https://github.com/databrickslabs/dqx/issues/609)). DQX now skip check evaluation if columns or filters are incorrect allowing other checks to proceed even if one rule fails. The DQX engine validates specified column, columns and filter fields against the input DataFrame before applying checks, skipping evaluation and providing informative error messages if any fields are invalid.
+* Updated user guide docs ([#607](https://github.com/databrickslabs/dqx/issues/607)). The documentation for quality checking and integration options has been updated to provide accurate and detailed information on supported types and approaches. Quality checking can be performed in-transit (pre-commit), validating data on the fly during processing, or at-rest, checking existing data stored in tables.
+* Improved build process ([#618](https://github.com/databrickslabs/dqx/issues/618)). The hatch version has been updated to 1.15.0 to avoid compatibility issues with click version 8.3 and later, which introduced a bug affecting hatch. Additionally, the project's dependencies have been updated, including bumping the `databricks-labs-pytester` version from 0.7.2 to 0.7.4, and code refactoring has been done to use a single Lakebase instance for all integration tests, with retry logic added to handle cases where the workspace quota limit for the number of Lakebase instances is exceeded, enhancing the testing infrastructure and improving test reliability. Furthermore, documentation updates have been made to clarify the application of quality checks to data using DQX. These changes aim to improve the efficiency, reliability, and clarity of the project's testing and documentation infrastructure.
+
 ## 0.9.3
 
 * Added support for running checks on multiple tables ([#566](https://github.com/databrickslabs/dqx/issues/566)). Added more flexibility and functionality in running data quality checks, allowing users to run checks on multiple tables in a single method call and as part of Workflows execution. Provided options to run checks for all configured run configs or for a specific run config, or for tables/views matching wildcard patterns. The CLI commands for running workflows have been updated to reflect and support these new functionalities. Additionally, new parameters have been added to configuration file to control the level of parallelism for these operations, such as `profiler_max_parallelism` and `quality_checker_max_parallelism`. A new demo has been added to showcases how to use the profiler and apply checks across multiple tables. The changes aim to improve scalability of DQX.

docs/dqx/docs/demos.mdx

@@ -8,22 +8,22 @@ import Admonition from '@theme/Admonition';
 
 Import the following notebooks in the Databricks workspace to try DQX out:
 ## Use as Library
-* [DQX Quick Start Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_quick_start_demo_library.py) - quickstart on how to use DQX as a library.
-* [DQX Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_library.py) - demonstrates how to use DQX as a library.
-* [DQX Demo Notebook for Data Quality Summary Metrics](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_summary_metrics.py) - demonstrates how to generate summary-level data quality metrics when validating data with DQX.
-* [DQX Demo Notebook for Profiling and Applying Checks at Scale on Multiple Tables](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_multi_table_demo.py) - demonstrates how to use DQX as a library at scale to apply checks on multiple tables.
-* [DQX Demo Notebook for Spark Structured Streaming (Native End-to-End Approach)](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_streaming_demo_native.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, using the built-in end-to-end method to handle both reading and writing.
-* [DQX Demo Notebook for Spark Structured Streaming (DIY Approach)](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_streaming_demo_diy.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, while handling reading and writing on your own outside DQX using Spark API.
-* [DQX Demo Notebook for Lakeflow Pipelines (formerly DLT)](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_dlt_demo.py) - demonstrates how to use DQX as a library with Lakeflow Pipelines.
-* [DQX Asset Bundles Demo](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_asset_bundle/README.md) - demonstrates how to use DQX as a library with Databricks Asset Bundles.
-* [DQX Demo for dbt](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_dbt/README.md) - demonstrates how to use DQX as a library with dbt projects.
+* [DQX Quick Start Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_quick_start_demo_library.py) - quickstart on how to use DQX as a library.
+* [DQX Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_library.py) - demonstrates how to use DQX as a library.
+* [DQX Demo Notebook for Data Quality Summary Metrics](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_summary_metrics.py) - demonstrates how to generate summary-level data quality metrics when validating data with DQX.
+* [DQX Demo Notebook for Profiling and Applying Checks at Scale on Multiple Tables](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_multi_table_demo.py) - demonstrates how to use DQX as a library at scale to apply checks on multiple tables.
+* [DQX Demo Notebook for Spark Structured Streaming (Native End-to-End Approach)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_streaming_demo_native.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, using the built-in end-to-end method to handle both reading and writing.
+* [DQX Demo Notebook for Spark Structured Streaming (DIY Approach)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_streaming_demo_diy.py) - demonstrates how to use DQX as a library with Spark Structured Streaming, while handling reading and writing on your own outside DQX using Spark API.
+* [DQX Demo Notebook for Lakeflow Pipelines (formerly DLT)](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_dlt_demo.py) - demonstrates how to use DQX as a library with Lakeflow Pipelines.
+* [DQX Asset Bundles Demo](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_asset_bundle/README.md) - demonstrates how to use DQX as a library with Databricks Asset Bundles.
+* [DQX Demo for dbt](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_dbt/README.md) - demonstrates how to use DQX as a library with dbt projects.
 
 ## Deploy as Workspace Tool
-* [DQX Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_tool.py) - demonstrates how to use DQX as a tool when installed in the workspace.
+* [DQX Demo Notebook](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_tool.py) - demonstrates how to use DQX as a tool when installed in the workspace.
 
 ## Use Cases
-* [DQX for PII Detection Notebook](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_pii_detection.py) - demonstrates how to use DQX to check data for Personally Identifiable Information (PII).
-* [DQX for Manufacturing Notebook](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_manufacturing_demo.py) - demonstrates how to use DQX to check data quality for Manufacturing Industry datasets.
+* [DQX for PII Detection Notebook](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_pii_detection.py) - demonstrates how to use DQX to check data for Personally Identifiable Information (PII).
+* [DQX for Manufacturing Notebook](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_manufacturing_demo.py) - demonstrates how to use DQX to check data quality for Manufacturing Industry datasets.
 
 <br />
 <Admonition type="tip" title="Execution Environment">

docs/dqx/docs/dev/contributing.mdx

@@ -114,7 +114,7 @@ In all cases, the full test suite is executed on code merged into `main` branch
 
 This section provides a step-by-step guide for setting up your local environment and dependencies for efficient development.
 
-To start with, install the required python version on your computer (see `python=<version>` field in [pyproject.toml](https://github.com/databrickslabs/dqx/blob/v0.9.3/pyproject.toml)).
+To start with, install the required python version on your computer (see `python=<version>` field in [pyproject.toml](https://github.com/databrickslabs/dqx/blob/v0.10.0/pyproject.toml)).
 On MacOSX, run:
 ```
 # double check the required version in pyproject.toml

docs/dqx/docs/guide/quality_checks_apply.mdx

@@ -892,7 +892,7 @@ Below is a sample output of a check stored in a result column (error or warning)
 ```
 
 The structure of the result columns is an array of struct containing the following fields
-(see the exact structure [here](https://github.com/databrickslabs/dqx/blob/v0.9.3/src/databricks/labs/dqx/schema/dq_result_schema.py)):
+(see the exact structure [here](https://github.com/databrickslabs/dqx/blob/v0.10.0/src/databricks/labs/dqx/schema/dq_result_schema.py)):
 - `name`: name of the check (string type).
 - `message`: message describing the quality issue (string type).
 - `columns`: name of the column(s) where the quality issue was found (string type).

docs/dqx/docs/reference/quality_checks.mdx

@@ -16,7 +16,7 @@ DQX provides a collection of predefined built-in quality rules (checks). Additio
 All DQX rules are evaluated independently and in parallel. If one rule fails, the others are still processed.
 All rule types, including row-level and dataset-level rules, can be defined and applied simultaneously in a single execution.
 
-You can explore the implementation details of the check functions [here](https://github.com/databrickslabs/dqx/blob/v0.9.3/src/databricks/labs/dqx/check_funcs.py).
+You can explore the implementation details of the check functions [here](https://github.com/databrickslabs/dqx/blob/v0.10.0/src/databricks/labs/dqx/check_funcs.py).
 
 ## Row-level checks reference
 
@@ -3077,7 +3077,7 @@ The built-in check supports several configurable parameters:
 DQX automatically installs the built-in entity recognition models at runtime if they are not already available.
 However, for better performance and to avoid potential out-of-memory issues, it is recommended to pre-install models using pip install.
 Any additional models used in a custom configuration must also be installed on your Databricks cluster.
-See the [Using DQX for PII Detection](https://github.com/databrickslabs/dqx/blob/v0.9.3/demos/dqx_demo_pii_detection.py) notebook for examples of custom model installation.
+See the [Using DQX for PII Detection](https://github.com/databrickslabs/dqx/blob/v0.10.0/demos/dqx_demo_pii_detection.py) notebook for examples of custom model installation.
 </Admonition>
 
 #### Using Built-in NLP Engine Configurations

pyproject.toml

@@ -6,6 +6,11 @@ description = 'Data Quality eXtended (DQX) is a Python library for data quality
 license-files = { paths = ["LICENSE", "NOTICE"] }
 requires-python = ">=3.10"
 keywords = ["Databricks"]
+authors = [
+    {name = "Databricks Labs", email = "[email protected]"},
+    { name = "Marcin Wojtyczka", email = "[email protected]" },
+    { name = "Alex Ott", email = "[email protected]" },
+]
 maintainers = [
     { name = "Databricks Labs", email = "[email protected]" },
     { name = "Marcin Wojtyczka", email = "[email protected]" },

src/databricks/labs/dqx/__about__.py

@@ -1 +1 @@
-__version__ = "0.9.3"
+__version__ = "0.10.0"