[BUG] Row-level filtering marking the records as pass when null values are present in the column

[eapframework]

I am working on filtering data based on row-level checks. It's working fine when notnull values are present in the column

But incorrectly marking the records as pass when null values are present in the column.

For example

 import sparkSession.implicits._

    Seq(
      (1, "a", 1),
      (2, "b", 3),
      (3, null, null),
      (4, "c", 5),
      (5, null, null),
      (6, "d", 7)
    ).toDF("item", "att1", "att2")

Applied below rules:

rule1 : .isPrimaryKey("att1","att2")
rule2: .isGreaterThan("att2", "att1")
rule3: .isgreaterthanorequalto("att2", "att1")
+----+----+----+-----+-----+-----+
|item|att1|att2|rule1|rule2|rule3|
+----+----+----+-----+-----+-----+
|   1|   a|   1| true|false| true|
|   2|   b|   3| true| true| true|
|   3|null|null| true| true| true|
|   4|   c|   5| true| true| true|
|   5|null|null| true| true| true|
|   6|   d|   7| true| true| true|
+----+----+----+-----+-----+-----+

When columns values are null, the row-level check status is considered as true but it should be false.

[github-actions]

This is a known behavior issue with how row-level results are computed for grouping-based analyzers (like isPrimaryKey/Uniqueness) and compliance-based checks (like isGreaterThan/isGreaterThanOrEqualTo).

For isPrimaryKey (Uniqueness): In GroupingAnalyzers.scala, the fullColumn is computed using a window count over the grouping columns. When values are NULL, Spark's count() over a window with NULL partition keys groups all NULLs together, but the NULL-filtering logic (atLeastOneNonNullGroupingColumn) should exclude all-null rows. However, the row-level column is evaluated on the original data using a window function, and rows where all grouping columns are NULL end up with a count that incorrectly marks them as unique.

For isGreaterThan/isGreaterThanOrEqualTo (Compliance): These use Compliance with a predicate like att2 > att1. When both columns are NULL, NULL > NULL evaluates to NULL in SQL. The conditionSelectionGivenColumn with FilteredRowOutcome.TRUE replaces filtered-out rows with true, but the default path returns the criterion directly — and NULL gets coerced to true downstream when combined with other checks via AND.

Workaround: Add an explicit completeness check or combine with a null-check filter:

.isComplete("att1")
.isComplete("att2")
.satisfies("att2 > att1", "rule2", _ == 1.0).where("att1 IS NOT NULL AND att2 IS NOT NULL")

Alternatively, use AnalyzerOptions with FilteredRowOutcome.NULL so that rows with NULLs are marked as null rather than true:

.satisfies("att2 > att1", "rule2", _ == 1.0,
  analyzerOptions = Some(AnalyzerOptions(filteredRow = FilteredRowOutcome.NULL)))

This is a design limitation — the row-level result framework doesn't currently distinguish between "not applicable (NULL inputs)" and "passed". A proper fix would require the Uniqueness/Compliance row-level columns to explicitly handle NULL inputs by returning false (or null) instead of allowing NULL propagation to be interpreted as true.

---

Generated by AI (model: us.anthropic.claude-opus-4-6-v1, prompt: 5609e400) — may not be fully accurate. Reply if this doesn't help.