-
Notifications
You must be signed in to change notification settings - Fork 69
Open
Labels
enhancementNew feature or requestNew feature or request
Description
Is there an existing issue for this?
- I have searched the existing issues
Problem statement
DQX supports built-in check functions for potentially large input objects (e.g. regex, JSON, or geometry checks on large string objects). The original column value is often embedded in DQX metadata columns by default. This may significantly increase the dataset size when checking columns with large objects.
Proposed Solution
Add a MetadataLevel class with the following options:
MetadataLevel.COMPACTto suppress input values from the DQX metadataMetadataLevel.STANDARDto use the existing behavior
Provide parameters for users to suppress input values in the DQX metadata:
- Add
metadata_levelto aDQRuleto control behavior for specific checks at the rule level - Add
metadata_levelto theDQEngineto control behavior for all checks at the engine level
The default will be MetadataLevel.STANDARD unless explicitly configured.
Additional Context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request