Feat add json validation checks #616

cornzyblack · 2025-10-15T22:40:13Z

Changes

Linked issues

Resolves #595

Tests

…attribute

Copilot

Pull Request Overview

Adds row-level JSON validation checks and integrates them into examples and tests.

Introduces is_valid_json and has_json_keys row checks.
Updates YAML examples, reference docs, and integration/unit tests to cover the new checks.

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/databricks/labs/dqx/check_funcs.py	Adds JSON validation/check functions; core logic for new checks.
tests/unit/test_build_rules.py	Extends metadata conversion tests to include new JSON checks.
tests/integration/test_apply_checks.py	Adds col_json_str to test schemas and values; exercises new checks in streaming and class-based tests.
tests/resources/all_row_checks.yaml	Includes is_valid_json check in the “all row checks” YAML.
src/databricks/labs/dqx/llm/resources/yaml_checks_examples.yml	Adds examples for is_valid_json and has_json_keys.
docs/dqx/docs/reference/quality_checks.mdx	Documents new checks and shows usage examples.

Comments suppressed due to low confidence (1)

docs/dqx/docs/reference/quality_checks.mdx:1

Both examples use the same name 'col_json_str_has_json_keys', which is confusing and may collide in practice. Use distinct, descriptive names (e.g., 'col_json_str_has_no_json_key1' and 'col_json_str_has_no_json_key1_key2').

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-16T09:19:24Z

src/databricks/labs/dqx/check_funcs.py

+    col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
+    return make_condition(
+        ~F.when(F.col(col_expr_str).isNotNull(), F.try_parse_json(col_expr_str).isNotNull()),
+        F.concat_ws(
+            "", F.lit("Value '"), col_expr.cast("string"), F.lit(f"' in Column '{col_expr_str}' is not a valid JSON")
+        ),
+        f"{col_str_norm}_is_not_valid_json",
+    )


is_valid_json dereferences the column by name (F.col(col_expr_str)) rather than using the resolved expression (col_expr). This breaks when the caller supplies a column expression (e.g., F.trim('c')), which may not exist as a named column. Use col_expr consistently in both the null and try_parse_json checks.

Copilot · 2025-10-16T09:19:24Z

src/databricks/labs/dqx/check_funcs.py

+
+    unique_keys_lit = F.lit(list(set(keys)))
+    col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
+    json_keys_array = F.json_object_keys(col_expr)


pyspark.sql.functions does not expose json_object_keys; this will raise AttributeError at runtime. Use F.expr to call the SQL function or derive keys via from_json + map_keys. For example: json_keys_array = F.expr(f"json_object_keys({col_expr_str})").

Suggested change

json_keys_array = F.json_object_keys(col_expr)

json_keys_array = F.expr(f"json_object_keys({col_expr_str})")

Copilot · 2025-10-16T09:19:25Z

src/databricks/labs/dqx/check_funcs.py

+
+    if require_all:
+        condition = F.size(F.array_except(unique_keys_lit, json_keys_array)) == 0
+    else:
+        condition = F.when(is_valid_json(col_str_norm).isNull(), F.arrays_overlap(json_keys_array, unique_keys_lit))


has_json_keys returns a null condition (and thus passes) for invalid JSON or null values, which means it won't flag missing keys unless a separate is_valid_json rule is also configured. To make has_json_keys self-contained, include a JSON-validity guard in the condition (so invalid JSON fails this check as well). For example: compute json_valid = F.try_parse_json(col_expr).isNotNull() and combine it with the key presence logic.

Suggested change

if require_all:

condition = F.size(F.array_except(unique_keys_lit, json_keys_array)) == 0

else:

condition = F.when(is_valid_json(col_str_norm).isNull(), F.arrays_overlap(json_keys_array, unique_keys_lit))

json_valid = F.try_parse_json(col_expr).isNotNull()

if require_all:

condition = json_valid & (F.size(F.array_except(unique_keys_lit, json_keys_array)) == 0)

else:

condition = json_valid & F.arrays_overlap(json_keys_array, unique_keys_lit)

Copilot · 2025-10-16T09:19:25Z

src/databricks/labs/dqx/check_funcs.py

+        if not isinstance(key, (str)):
+            raise InvalidParameterError("All keys must be of type string.")
+
+    unique_keys_lit = F.lit(list(set(keys)))


Using set(keys) loses ordering and can lead to non-deterministic error messages (and test flakiness). Prefer a stable order: unique_keys_lit = F.lit(sorted(set(keys))).

Suggested change

unique_keys_lit = F.lit(list(set(keys)))

unique_keys_lit = F.lit(sorted(set(keys)))

Copilot · 2025-10-16T09:19:25Z

src/databricks/labs/dqx/check_funcs.py

+            F.concat_ws(", ", F.lit(keys)),
+            F.lit("]"),
+        ),
+        f"{col_str_norm}_has_no_json_keys",


The alias for has_json_keys does not include the target keys. If this rule is applied multiple times to the same column with different key sets, the resulting alias collides (e.g., duplicate 'col_json_str_has_no_json_keys'). Include the keys (and optionally require_all) in the alias to ensure uniqueness, e.g., alias_name = f"{col_str_norm}has_no_json{'_'.join(sorted(set(keys)))}".

Suggested change

f"{col_str_norm}_has_no_json_keys",

f"{col_str_norm}_has_no_json_{'_'.join(sorted(set(keys)))}{'_all' if require_all else '_any'}",

Copilot · 2025-10-16T09:19:26Z

docs/dqx/docs/reference/quality_checks.mdx

+| `is_valid_json`                    | Checks whether the values in the input column are valid JSON objects.                                                                                                                                                                                                                                                                                                                                                             | `column`: column to check (can be a string column name or a column expression)                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `has_json_keys`                    | Checks whether the values in the input column contain specific JSON keys.                                                                                                                                                                                                                                                                                                                                                         | `column`: column to check (can be a string column name or a column expression); `keys`: list of JSON keys to check for                                                                                                                                                                                                                                                                                                                                                                                                                               |


Documentation gaps: (1) is_valid_json validates any JSON (object/array/etc.), not only 'JSON objects'—please reword. (2) has_json_keys supports a require_all flag (default True) but it's not documented—add it to the parameter list.

Suggested change

| `is_valid_json` | Checks whether the values in the input column are valid JSON objects. | `column`: column to check (can be a string column name or a column expression) |

| `has_json_keys` | Checks whether the values in the input column contain specific JSON keys. | `column`: column to check (can be a string column name or a column expression); `keys`: list of JSON keys to check for |

| `is_valid_json` | Checks whether the values in the input column are valid JSON (objects, arrays, strings, numbers, etc.), not just JSON objects. | `column`: column to check (can be a string column name or a column expression) |

| `has_json_keys` | Checks whether the values in the input column contain specific JSON keys. | `column`: column to check (can be a string column name or a column expression); `keys`: list of JSON keys to check for; `require_all`: whether all keys must be present (default: True) |

Copilot · 2025-10-16T09:19:26Z

src/databricks/labs/dqx/llm/resources/yaml_checks_examples.yml

+      keys:
+      - key1
+- criticality: error
+  name: col_json_str_has_no_json_keys2


The example name 'col_json_str_has_no_json_keys2' is ambiguous. Consider aligning with the keys being checked (e.g., 'col_json_str_has_no_json_key1_key2') for clarity and consistency with other examples.

Suggested change

name: col_json_str_has_no_json_keys2

name: col_json_str_has_no_json_key1_key2

src/databricks/labs/dqx/check_funcs.py

mwojtyczka

Please add performance and integration tests for the new check functions. The original ticket also mentioned has_json_schema

mwojtyczka · 2025-10-16T09:20:36Z

docs/dqx/docs/reference/quality_checks.mdx

 | `is_valid_date`                    | Checks whether the values in the input column have valid date formats.                                                                                                                                                                                                                                                                                                                                                            | `column`: column to check (can be a string column name or a column expression); `date_format`: optional date format (e.g. 'yyyy-mm-dd')                                                                                                                                                                                                                                                                                                                                                                                                              |
 | `is_valid_timestamp`               | Checks whether the values in the input column have valid timestamp formats.                                                                                                                                                                                                                                                                                                                                                       | `column`: column to check (can be a string column name or a column expression); `timestamp_format`: optional timestamp format (e.g. 'yyyy-mm-dd HH:mm:ss')                                                                                                                                                                                                                                                                                                                                                                                           |
+| `is_valid_json`                    | Checks whether the values in the input column are valid JSON objects.                                                                                                                                                                                                                                                                                                                                                             | `column`: column to check (can be a string column name or a column expression)                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+| `has_json_keys`                    | Checks whether the values in the input column contain specific JSON keys.                                                                                                                                                                                                                                                                                                                                                         | `column`: column to check (can be a string column name or a column expression); `keys`: list of JSON keys to check for                                                                                                                                                                                                                                                                                                                                                                                                                               |


require_all misssing

mwojtyczka · 2025-10-16T09:22:04Z

docs/dqx/docs/reference/quality_checks.mdx

      name="col6_is_not_valid_timestamp2"
    ),

+    # is_valid_json check


pls group examples for has_json_keys and is_valid_json together

mwojtyczka · 2025-10-16T09:26:17Z

src/databricks/labs/dqx/check_funcs.py

+    Checks whether the values in the input column contain specific JSON keys.
+
+    Args:
+        column (str | Column): The name of the column or the column itself to check for JSON keys.


Suggested change

column (str | Column): The name of the column or the column itself to check for JSON keys.

column (str | Column): The name of the column or the column expression to check for JSON keys.

mwojtyczka · 2025-10-16T09:27:25Z

tests/resources/all_row_checks.yaml

+    function: is_valid_json
+    arguments:
+      column: col_json_str
+


add has_json_keys as well

cornzyblack · 2025-10-17T08:04:01Z

Updates:

WIP: Still working on has_json_schema
Added unit, integration and performance tests for is_valid_json and has_json_keys (grouped them)

Notes

Unit tests for has_valid_json_schema might not be possible (see image below) as it returns an error when I ran it, suspecting it is because _expected_schema = _get_schema(schema) uses types.StructType.fromDDL(input_schema), which requires a Spark session.
I renamed has_json_schema to has_valid_json_schema as I feel that sounds more descriptive. (Happy to revert when it is done if needed).

mwojtyczka · 2025-10-18T21:41:05Z

has_valid_json_schema

if you need a spark session, you have to use integration tests.
Agree, has_valid_json_schema is better

ghanse · 2025-10-20T00:52:18Z

src/databricks/labs/dqx/check_funcs.py

+        raise InvalidParameterError("All keys must be of type string.")
+
+    col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column)
+    json_keys_array = F.json_object_keys(col_expr)


Interestingly, json_object_keys will only return keys of the outer object. This is probably fine, but we should document it explicitly.

…ack/dqx into feat-add-json-validation-checks

cornzyblack added 27 commits July 17, 2025 18:41

move criticiality of rule inro _validate_attributes

6cd9f02

since criticality is validated after creation, filter by criticality …

98371b7

…attribute

Merge branch 'main' of github.com:cornzyblack/dqx

1a58e16

Merge branch 'main' of github.com:cornzyblack/dqx

98803bc

Merge branch 'main' of github.com:cornzyblack/dqx

acf3767

Merge branch 'main' of github.com:cornzyblack/dqx

0766f2b

Merge branch 'main' of github.com:cornzyblack/dqx

3d0fd34

Merge branch 'main' of github.com:cornzyblack/dqx

e5712fc

Merge branch 'main' of github.com:cornzyblack/dqx

9bf6d98

Merge branch 'main' of github.com:cornzyblack/dqx

1e4d783

Merge branch 'main' of github.com:cornzyblack/dqx

fcdb1ce

Merge branch 'main' of github.com:cornzyblack/dqx

2393404

Merge branch 'main' of github.com:cornzyblack/dqx

eddc874

Merge branch 'main' of github.com:cornzyblack/dqx

c378b6d

Merge branch 'main' of github.com:cornzyblack/dqx

cb6f9ef

feat: add check for valid json

82c7a22

feat: add checks for is_valid_json

f1ec4af

feat: add is_valid_json

dfa9649

feat: add has_json_keys

89f2811

refactor: change logic

02466c1

refactor: invert

ccb6e05

refactor: negate

156a9c2

refactor: update

8d30ff6

refactor: update

1873d72

refactor: update

5109c27

refactor: updates

ceecf7d

refactor: change and update

0c94089

mwojtyczka requested a review from Copilot October 16, 2025 09:14

Merge branch 'main' into feat-add-json-validation-checks

246833b

Copilot AI reviewed Oct 16, 2025

View reviewed changes

mwojtyczka requested changes Oct 16, 2025

View reviewed changes

cornzyblack added 7 commits October 16, 2025 11:48

refactor: fix docs

05365e0

refactor: updates

7be64e6

refactor: update logic

c7d8406

refactor: explcit True

66cbb13

refactor: remove repetition

70e19bd

refactor: remove as it depends on spark

a168d64

feat: add perf test for 2 tests (remaining 1)

c3c23e7

Merge branch 'main' into feat-add-json-validation-checks

984bbb8

ghanse reviewed Oct 20, 2025

View reviewed changes

cornzyblack and others added 12 commits October 21, 2025 07:30

refactor: switch back to has_json_schema

3e63312

Merge branch 'feat-add-json-validation-checks' of github.com:cornzybl…

9ed893a

…ack/dqx into feat-add-json-validation-checks

docs: document properly that function only checks outside keys

a72bdb1

refactor: comment out to test

b8505e4

refactor: try using transform for strict comparison

a177c01

feat: implement changes

e0c3438

Merge branch 'main' into feat-add-json-validation-checks

853c8c0

format and add tests

3b0fd52

Merge branch 'feat-add-json-validation-checks' of github.com:cornzybl…

44881fe

…ack/dqx into feat-add-json-validation-checks

refactor: add to markdown

7b19d00

updates

96cbc8e

Merge branch 'main' into feat-add-json-validation-checks

0ff6ccb

	json_keys_array = F.json_object_keys(col_expr)
	json_keys_array = F.expr(f"json_object_keys({col_expr_str})")

	unique_keys_lit = F.lit(list(set(keys)))
	unique_keys_lit = F.lit(sorted(set(keys)))

	f"{col_str_norm}_has_no_json_keys",
	f"{col_str_norm}_has_no_json_{'_'.join(sorted(set(keys)))}{'_all' if require_all else '_any'}",

		\| `is_valid_json` \| Checks whether the values in the input column are valid JSON objects. \| `column`: column to check (can be a string column name or a column expression) \|
		\| `has_json_keys` \| Checks whether the values in the input column contain specific JSON keys. \| `column`: column to check (can be a string column name or a column expression); `keys`: list of JSON keys to check for \|

	name: col_json_str_has_no_json_keys2
	name: col_json_str_has_no_json_key1_key2

	column (str \| Column): The name of the column or the column itself to check for JSON keys.
	column (str \| Column): The name of the column or the column expression to check for JSON keys.

Feat add json validation checks #616

Are you sure you want to change the base?

Feat add json validation checks #616

Uh oh!

Conversation

cornzyblack commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Linked issues

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mwojtyczka left a comment

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

cornzyblack commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mwojtyczka commented Oct 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghanse Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cornzyblack commented Oct 15, 2025 •

edited

Loading

cornzyblack commented Oct 17, 2025 •

edited

Loading

mwojtyczka commented Oct 18, 2025 •

edited

Loading