Feat add json validation checks #616
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -1747,6 +1747,69 @@ def apply(df: DataFrame) -> DataFrame: | |||||||||||||||||||||||
| return condition, apply | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| @register_rule("row") | ||||||||||||||||||||||||
| def is_valid_json(column: str | Column) -> Column: | ||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||
| Checks whether the values in the input column is a valid JSON string. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Args: | ||||||||||||||||||||||||
| column: Column name (str) or Column expression to check for valid JSON. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Returns: | ||||||||||||||||||||||||
| A Spark Column representing the condition for invalid JSON strings. | ||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||
| col_str_norm, col_expr_str, col_expr = _get_normalized_column_and_expr(column) | ||||||||||||||||||||||||
| return make_condition( | ||||||||||||||||||||||||
| ~F.when(F.col(col_expr_str).isNotNull(), F.try_parse_json(col_expr_str).isNotNull()), | ||||||||||||||||||||||||
ghanse marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||||||||||
| F.concat_ws( | ||||||||||||||||||||||||
| "", F.lit("Value '"), col_expr.cast("string"), F.lit(f"' in Column '{col_expr_str}' is not a valid JSON") | ||||||||||||||||||||||||
| ), | ||||||||||||||||||||||||
| f"{col_str_norm}_is_not_valid_json", | ||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||
|
Comment on lines
1761
to
1771
There was a problem hiding this comment. is_valid_json dereferences the column by name (F.col(col_expr_str)) rather than using the resolved expression (col_expr). This breaks when the caller supplies a column expression (e.g., F.trim('c')), which may not exist as a named column. Use col_expr consistently in both the null and try_parse_json checks. |
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| @register_rule("row") | ||||||||||||||||||||||||
| def has_json_keys(column: str | Column, keys: list[str], require_all: bool = True) -> Column: | ||||||||||||||||||||||||
| """ | ||||||||||||||||||||||||
| Checks whether the values in the input column contain specific JSON keys. | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Args: | ||||||||||||||||||||||||
| column (str | Column): The name of the column or the column itself to check for JSON keys. | ||||||||||||||||||||||||
|
||||||||||||||||||||||||
| column (str | Column): The name of the column or the column itself to check for JSON keys. | |
| column (str | Column): The name of the column or the column expression to check for JSON keys. |
Outdated
There was a problem hiding this comment.
Using set(keys) loses ordering and can lead to non-deterministic error messages (and test flakiness). Prefer a stable order: unique_keys_lit = F.lit(sorted(set(keys))).
| unique_keys_lit = F.lit(list(set(keys))) | |
| unique_keys_lit = F.lit(sorted(set(keys))) |
There was a problem hiding this comment.
pyspark.sql.functions does not expose json_object_keys; this will raise AttributeError at runtime. Use F.expr to call the SQL function or derive keys via from_json + map_keys. For example: json_keys_array = F.expr(f"json_object_keys({col_expr_str})").
| json_keys_array = F.json_object_keys(col_expr) | |
| json_keys_array = F.expr(f"json_object_keys({col_expr_str})") |
There was a problem hiding this comment.
Interestingly, json_object_keys will only return keys of the outer object. This is probably fine, but we should document it explicitly.
Outdated
There was a problem hiding this comment.
has_json_keys returns a null condition (and thus passes) for invalid JSON or null values, which means it won't flag missing keys unless a separate is_valid_json rule is also configured. To make has_json_keys self-contained, include a JSON-validity guard in the condition (so invalid JSON fails this check as well). For example: compute json_valid = F.try_parse_json(col_expr).isNotNull() and combine it with the key presence logic.
| if require_all: | |
| condition = F.size(F.array_except(unique_keys_lit, json_keys_array)) == 0 | |
| else: | |
| condition = F.when(is_valid_json(col_str_norm).isNull(), F.arrays_overlap(json_keys_array, unique_keys_lit)) | |
| json_valid = F.try_parse_json(col_expr).isNotNull() | |
| if require_all: | |
| condition = json_valid & (F.size(F.array_except(unique_keys_lit, json_keys_array)) == 0) | |
| else: | |
| condition = json_valid & F.arrays_overlap(json_keys_array, unique_keys_lit) |
Outdated
There was a problem hiding this comment.
The alias for has_json_keys does not include the target keys. If this rule is applied multiple times to the same column with different key sets, the resulting alias collides (e.g., duplicate 'col_json_str_has_no_json_keys'). Include the keys (and optionally require_all) in the alias to ensure uniqueness, e.g., alias_name = f"{col_str_norm}has_no_json{'_'.join(sorted(set(keys)))}".
| f"{col_str_norm}_has_no_json_keys", | |
| f"{col_str_norm}_has_no_json_{'_'.join(sorted(set(keys)))}{'_all' if require_all else '_any'}", |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -189,6 +189,28 @@ | |||||
| arguments: | ||||||
| column: col5 | ||||||
| date_format: yyyy-MM-dd | ||||||
| - criticality: error | ||||||
| check: | ||||||
| function: is_valid_json | ||||||
| arguments: | ||||||
| column: col_json_str | ||||||
| - criticality: error | ||||||
| check: | ||||||
| function: has_json_keys | ||||||
| arguments: | ||||||
| column: col_json_str | ||||||
| keys: | ||||||
| - key1 | ||||||
| - criticality: error | ||||||
| name: col_json_str_has_no_json_keys2 | ||||||
|
||||||
| name: col_json_str_has_no_json_keys2 | |
| name: col_json_str_has_no_json_key1_key2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation gaps: (1) is_valid_json validates any JSON (object/array/etc.), not only 'JSON objects'—please reword. (2) has_json_keys supports a require_all flag (default True) but it's not documented—add it to the parameter list.