Skip to content

Commit 5162364

Browse files
committed
Added a null-aware anti-join dynamic-filter regression test.
Exercises the pushdown path the existing in-memory tests miss: parquet with row-level filtering, so the pushed dynamic filter actually drops rows. Without the fix `id NOT IN (SELECT eid ...)` returns 1 and 3 instead of zero rows.
1 parent a71cb76 commit 5162364

1 file changed

Lines changed: 65 additions & 0 deletions

File tree

datafusion/sqllogictest/test_files/null_aware_anti_join.slt

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -451,3 +451,68 @@ DROP TABLE customers_test;
451451

452452
statement ok
453453
DROP TABLE all_null_banned;
454+
455+
#############
456+
## Test: dynamic filter pushdown must not drop the inner NULL.
457+
## With join dynamic filter pushdown on, the build-side filter pushed to the probe scan would drop
458+
## the inner NULL, but NOT IN three-valued logic needs it to collapse the result to zero rows. The
459+
## in-memory VALUES scans above never apply the pushed filter, so this case needs a parquet scan.
460+
#############
461+
462+
statement ok
463+
set datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
464+
465+
# Row-level parquet filtering, so the pushed filter actually drops matching rows instead of only
466+
# pruning row groups. Without this the single row group is read whole and the NULL never gets dropped.
467+
statement ok
468+
set datafusion.execution.parquet.pushdown_filters = true;
469+
470+
statement ok
471+
CREATE TABLE asa_outer(id INT) AS VALUES (1), (2), (3);
472+
473+
statement ok
474+
CREATE TABLE asa_inner(eid INT) AS VALUES (2), (NULL);
475+
476+
query I
477+
COPY asa_outer TO 'test_files/scratch/null_aware_anti_join/asa_outer.parquet' STORED AS PARQUET;
478+
----
479+
3
480+
481+
query I
482+
COPY asa_inner TO 'test_files/scratch/null_aware_anti_join/asa_inner.parquet' STORED AS PARQUET;
483+
----
484+
2
485+
486+
statement ok
487+
CREATE EXTERNAL TABLE asa_outer_parquet(id INT)
488+
STORED AS PARQUET
489+
LOCATION 'test_files/scratch/null_aware_anti_join/asa_outer.parquet';
490+
491+
statement ok
492+
CREATE EXTERNAL TABLE asa_inner_parquet(eid INT)
493+
STORED AS PARQUET
494+
LOCATION 'test_files/scratch/null_aware_anti_join/asa_inner.parquet';
495+
496+
# Expected: zero rows. Before the fix the pushed dynamic filter dropped the inner NULL, so the join
497+
# wrongly returned id = 1 and id = 3.
498+
query I
499+
SELECT id FROM asa_outer_parquet WHERE id NOT IN (SELECT eid FROM asa_inner_parquet) ORDER BY id;
500+
----
501+
502+
statement ok
503+
DROP TABLE asa_outer;
504+
505+
statement ok
506+
DROP TABLE asa_inner;
507+
508+
statement ok
509+
DROP TABLE asa_outer_parquet;
510+
511+
statement ok
512+
DROP TABLE asa_inner_parquet;
513+
514+
statement ok
515+
set datafusion.execution.parquet.pushdown_filters = false;
516+
517+
statement ok
518+
set datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;

0 commit comments

Comments
 (0)