Skip to content

perf(table): prune parquet row groups by dictionary - #2006

Open
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-dictionary-row-group-pruning
Open

perf(table): prune parquet row groups by dictionary#2006
fallintoplace wants to merge 3 commits into
apache:mainfrom
fallintoplace:perf/parquet-dictionary-row-group-pruning

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Use Parquet dictionaries during row-group pruning.

  • EqualTo and IN predicates can skip dictionary-only row groups.
  • Keeps the current stats and Bloom filter checks.
  • Runs the dictionary check between stats and Bloom filters.
  • Only trusts a dictionary when encoding stats show no plain fallback pages.
  • Keeps the row group for missing metadata, unsupported types, malformed pages, or read errors.
  • Groups dictionary predicates once per scan instead of once per row group.

Why

Min and max stats can be too broad for low-cardinality columns. A complete dictionary can prove that none of the requested values exist, so the row group does not need to be read.

Tests

  • GOMAXPROCS=2 go test -p 1 -vet=off ./table/internal ./table -count=1
  • Focused dictionary, Bloom filter, scan-planning, and malformed-page tests

Benchmark

Command:

GOMAXPROCS=2 go test -p 1 -vet=off ./table/internal -run '^$' -bench '^BenchmarkDictionaryRowGroupPruning$' -benchtime=1s -count=3

Workload: 16 row groups with 4,096 low-cardinality INT32 values each. The benchmark covers an absent target that prunes every group and a target present in one group.

  • Without dictionary: 1.38 to 1.61 ms/op, about 1.62 MB/op.
  • With dictionary, target absent: 0.65 to 0.81 ms/op, about 0.63 MB/op.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant