[FEATURE]: Generate DQX rules from ODCS data contracts

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Open Data Contract Standard (ODCS) is becoming quite popular. This is a contract, so does not run rules as such. DQX rules can be generated for the contracts.

Proposed Solution

Converter to generated DQX rules from ODCS contract.

Example:
https://github.com/bitol-io/open-data-contract-standard/blob/main/docs/examples/quality/column-accuracy.odcs.yaml

Docs:
https://bitol-io.github.io/open-data-contract-standard/v3.0.2/

Additional Context

Requested by some customers and Ontos

[vb-dbrks]

Would love to pick this up :)

[vb-dbrks]

@mwojtyczka Based on discussions today with you and the Astellas team. Please let me know if anything here is unclear or not aligned.

What’s the need/requirement?

Customers implementing federated governance strategies need a standardised approach for defining data contracts for their Data Products. The Open Data Contract Standard (ODCS), part of the Linux Foundation, provides a comprehensive specification format that captures data product descriptors and properties.

E.g. Astellas requires the ability to generate DQ checks from ODCS contracts:

• Implicitly - Auto-derived from schema properties and logical types
• Explicitly - Defined within the contract's quality section

Proposed Capability

1. Implicitly Derived DQ Rules
DQ rules should be automatically inferred from ODCS schema properties, including:

• Logical Types: Parse and derive checks based on ODCS logical type definitions
• Schema Constraints: Extract validation rules from property definitions
• Format Specifications: Generate format validation checks

Example Mappings:
ODCS Property -> Description -> DQX Check
unique: true -> Primary key/unique constraint -> is_unique
required: true -> Non-null constraint -> is_not_null
minValue/maxValue -> Range constraints -> is_in_range
minItems/maxItems -> Array size constraints -> Custom array validation
pattern: -> Regex pattern -> regex_match
format: -> Date/timestamp format -> is_valid_date/is_valid_timestamp

2. Explicitly Defined DQ Rules
ODCS supports multiple DQ rule definition approaches (see ODCS Data Quality section):

• Text-based expectations - Natural language rules (can leverage LLM-to-DQX converter)
• ODCS DQ Library - Standard library format rules
• SQL-based checks - Direct SQL expressions (maps to sql_expression in DQX)
• Custom properties - External tool formats (including DQX native format)

Design Principles

All generated DQ rules must:

1. Be traceable to the source contract - Track ODCS contract version, name, and metadata
2. Inherit scheduling from ODCS - Use scheduling defined in the contract
3. Include contract versioning - Capture ODCS contract version in generated rules metadata
4. Persist results with lineage - Save DQ check results alongside associated ODCS contract metadata (version, name, descriptors).

Side note: Versioning DQRules is Out of scope for this feature, and will be implemented as a part another; but it will have to be included later in the lineage ODCS Version -> DQ Check Version -> Results/Metrics Summary

[mwojtyczka]

Thank you for the summary. Happy to see the customer contributing to this as well. For versioning added another ticket.