[FEATURE]: Data set level rules

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

No response

Proposed Solution

Whole data set level checks and window based checks, constraint checks like primary key.

Example checks:

• row count not less than
• row count not greater than
• row count equal
• column match ordered lisgt
• pk check (duplicate)
• outliers
• compound columns unique
• column /mean/median/avg not less/greater than
• ability to apply the checks on groups of rows

Additional Context

Similar to scala implementation

[Palkers76]

could detect FK to PK , i.e. relationship between fact and dim/dict tables

[mwojtyczka]

Name proposal for this new type of rule: DQDatasetRule