Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Problem statement
There are no performance benchmarks available for profiler and quality checks.
DQX is using native spark apis for quality checking so the impact is minimal. Nevertheless, it would be useful to have performance benchmarks.
Proposed Solution
Create performance benchmarks on how the performance is impacted by running profiler and applying quality rules on bigger data sets.
This should be written as performance tests that can be run weekly or monthly.
Additional Context
No response