Skip to content

Fix mis-pruned data files#1170

Open
dentiny wants to merge 1 commit into
duckdb:v1.5-variegatafrom
dentiny:hjiang/fix-mispruned-data-file
Open

Fix mis-pruned data files#1170
dentiny wants to merge 1 commit into
duckdb:v1.5-variegatafrom
dentiny:hjiang/fix-mispruned-data-file

Conversation

@dentiny
Copy link
Copy Markdown
Member

@dentiny dentiny commented May 18, 2026

Closes #1135

The bug is: when we get data files, current implementation only select those with columns which matches the predicate (i.e., col_b <= 42), but miss all data files without the requested column.

Data file query before this fix:

data.data_file_id IN (
	SELECT data_file_id FROM col_2_stats
	WHERE (value_count IS NULL OR value_count > 0)
	  AND (min_value IS NULL OR max_value IS NULL OR (min_value <= '42' AND max_value >= '42'))
  )

After the fix

(
	data.data_file_id NOT IN (SELECT data_file_id FROM col_2_stats)
	OR
	data.data_file_id IN (
	  SELECT data_file_id FROM col_2_stats
	  WHERE (value_count IS NULL OR value_count > 0)
		AND (min_value IS NULL OR max_value IS NULL OR (min_value <= '42' AND max_value >= '42'))
	)
  )

@dentiny dentiny force-pushed the hjiang/fix-mispruned-data-file branch from 2afa331 to 2ef5659 Compare May 18, 2026 17:27
@dentiny dentiny force-pushed the hjiang/fix-mispruned-data-file branch from 2ef5659 to 30ea0f5 Compare May 18, 2026 18:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant