[FEATURE]: Add severity level to the quality checks

[ZubeyirOflaz]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

We frequently encounter data quality errors which signify the subgroup of a given data cannot be relied upon. As an example, certain sensor readings indicate that the test step that contains the erroneous reading shouldn't be trusted.

For these type of use cases, it would be ideal to be able to set a severity level for the check, and specify a list of columns to group the data . If an error is encountered, then all the rows in the given subgroup is marked/quarantined

Proposed Solution

• Addition of a new optional parameter for the DQEngine to specify the list of columns for grouping.
• Addition of a new boolean field to the dq rules that will let users specify if an error is severe

If grouping columns are provided and if the dq rule is set to severe, any error that is detected would result in the whole subgroup to be quarantined or marked.

Additional Context

We've made an in house data quality framework where we provide this funcitonality. If you start accepting external contributors, please notify me and I would be happ to try and contribute the implementation for this to your library.

[mwojtyczka]

@ZubeyirOflaz we accept external contributors. Happy to see the PR.

We already have the severity levels: error and warning. Any reason we need a new concept for severity?

Is the following what you're looking for?

Input df:

id | sensor | val
-------------------
1  |  1     | 1
2  |  1     | 0
3  |  1     | 3

Let's say we have a check to validate that val > 0. If any of the values are 0 or below then all rows belonging to this sensor / group (1) should be quarantined.

quarantined df:

id | sensor | val | error
-------------------------------
1  |  1     | 1   | sensor failed group check
2  |  1     | 0   | sensor val not greater than 0
3  |  1     | 3   | sensor failed group check

[ZubeyirOflaz]

Hey @mwojtyczka

Thank you for the response and sorry for the late response. Yes what I would look for would be something similar to what you proposed in your example, though the error message doesn't need to be different between the observed instance that failed the check and the other group members.

If I understood the documentation, what you've shown in the example would also be possible with dqx if I do a second check to determine the group level failures, but this seemed to me as it would be inefficient compared to being able to make group level checks during the initial checks as well. The reason for my assumption is during the second check I would have to make an array search to see if I see any error messages with a higher severity level and then assign the error to the group. However, I might be wrong about this and if I am wrong we can close this as resolved.

I am happy to see that you now accept external contributors. If you think the addition of group level checks would be a meaningful improvement in performance for my type of use case I would be happy to try and contribute. I will look into this library in detail and see if I can implement our solution within dqx as well.

[mwojtyczka]

@ZubeyirOflaz We are happy to accept your contribution. Feel free to create PR for this.

The easiest way to handle this would be to add new extra_params to DQEngine like group_col_name: str | Column | None = None. This would be applicable to all checks which is probably what you need. If any of the checks fail, all other items belonging to the same group would fail.