Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Problem statement
When DQX is installed in the workspace, we provide profiling workflow/job to generate quality rule candidates. After that user can use the generated checks to do quality checks. However, performing the actual quality checking requires code level integration. In some cases, users may want to just do quality checking on the existing input data without performing any transformations.
Proposed Solution
Add quality checks workflow/job, that can be run to assess data quality of a dataset and write out quarantine and output to locations as specified in the config. This won't give the possibility to perform any data transformations as part of the quality checking but it would be useful in a lot of scenarios.
User journey:
- (option) Profiler job to profile the data and generate quality rule candidates
- Run Quality Check job to assess data quality of the input data, and persist the results as per config
- Refresh quality dashboard
Additional Context
No response