Track Value constraints through plan construction #708
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
oerling
added a commit
to oerling/verax
that referenced
this pull request
Dec 14, 2025
) Summary: Pull Request resolved: facebookincubator#708 Differential Revision: D89130357
oerling
force-pushed
the
export-D89130357
branch
from
f220662 to
d03c49c
Compare
oerling
added a commit
to oerling/verax
that referenced
this pull request
Dec 15, 2025
) Summary: Pull Request resolved: facebookincubator#708 Differential Revision: D89130357
oerling
force-pushed
the
export-D89130357
branch
from
d03c49c to
82e5bf5
Compare
added 2 commits
January 13, 2026 17:23
Summary: This diff adds support for exposing partition-level statistics from the Prism connector, enabling the optimizer to make better decisions for partitioned tables. The implementation focuses on the LocalHive connector and introduces a new `getPartitionStatistics()` API that returns detailed statistics for specific partitions and columns. The key motivation is to provide per-partition statistics (row counts, file counts, column statistics including min/max, null percentages, distinct counts, etc.) that can be used for query optimization, particularly for pruning partitions and estimating cardinalities. Previously, only table-level statistics were available. Main changes: 1. Added `PartitionStatistics` struct to ConnectorMetadata.h that holds per-partition statistics including numRows, numFiles, column names, and per-column statistics 2. Added `getPartitionStatistics()` virtual method to ConnectorSplitManager interface that takes a span of partition handles and column names and returns statistics for each partition 3. Enhanced `PartitionHandle` base class with `partition()` method to return partition path string in Hive format (e.g., "ds=2023-01-01/product=p1") 4. Implemented `HivePartitionHandle::makePartitionString()` to create Hive-formatted partition strings from partition key maps 5. Implemented partition filtering in `LocalHiveSplitManager::listPartitions()` to support partition pruning based on filters in the table handle - extracts filters that apply to partition columns and tests partition values against them 6. Implemented `LocalHiveSplitManager::getPartitionStatistics()` to return pre-computed statistics from LocalHivePartitionHandle objects 7. Added helper functions for converting partition values (stored as strings) to typed values and testing them against filters (testPartitionValue, makePartitionVector) 8. Enhanced `LocalHiveTableLayout::gatherFileStatistics()` to collect both data column and partition column statistics by: - Separating data and partition columns - Reading data column statistics from data files using existing file readers - Computing partition column statistics from partition value lists - Merging statistics and storing them per partition 9. Added `ColumnStatistics::toString()` method for debugging and logging 10. Changed mutex type from `std::mutex` to `std::recursive_mutex` in LocalHiveConnectorMetadata to support nested locking scenarios 11. Added `discretePredicateColumns()` override in LocalHiveTableLayout to expose partition columns for partition pruning The implementation reads statistics from data files (Parquet, ORC, etc.) for data columns and computes statistics for partition columns by examining the partition values. All statistics are pre-computed during table metadata initialization and cached in LocalHivePartitionHandle objects, making the getPartitionStatistics() call very fast. Differential Revision: D88694200
Summary: This diff introduces a comprehensive filter selectivity estimation framework for the Axiom optimizer. Filter selectivity estimation is crucial for query optimization as it enables accurate cardinality estimation after filters are applied, which directly impacts join ordering, physical operator selection, and overall query plan quality. The implementation provides sophisticated selectivity estimation for various predicate types including comparisons, ranges, IN clauses, AND/OR combinations, and null checks. The framework leverages column statistics (min/max values, cardinalities, null fractions) to compute mathematically sound selectivity estimates with proper NULL semantics handling. Main changes: 1. **New Filters.cpp/Filters.h (1,400+ lines)**: Core selectivity estimation framework with the following key functions: - `exprSelectivity()`: Entry point that dispatches to appropriate selectivity estimator based on expression type - `conjunctsSelectivity()`: Combines selectivity of AND predicates using probability multiplication with proper NULL handling - `combineConjuncts()`: Computes P(TRUE) and P(NULL) for conjunctions using formula: P(TRUE) = ∏ trueFractions, P(NULL) = ∏(trueFraction + nullFraction) - P(TRUE) - `combineDisjuncts()`: Computes selectivity for OR predicates using: P(TRUE) = 1 - ∏(1 - trueFraction) - `comparisonSelectivity()`: Estimates selectivity for equality and inequality comparisons between columns - `columnComparisonSelectivity()`: Template-based implementation for type-specific comparison selectivity using range overlap analysis - `rangeSelectivity()`: Estimates selectivity for range predicates (>, <, >=, <=) by computing intersection of constraint ranges - `inSelectivity()`: Handles IN clause selectivity by intersecting IN lists and counting matching values - Support for constraint propagation through `updateConstraints` parameter 2. **Comparison selectivity algorithm**: For comparing two columns a and b with ranges [al, ah] and [bl, bh] and cardinalities ac and bc: - For equality (a = b): Assumes uniform distribution within ranges. Computes overlap range, estimates distinct values in overlap for each column, takes minimum as matching values, divides by (ac × bc) - For inequality (a < b): Computes fraction of a's range below b using continuous approximation and integral over overlap region - Handles NULL semantics: comparisons with NULL yield NULL, all probabilities scaled by (1 - P(either null)) - Returns `Selectivity` struct with both `trueFraction` and `nullFraction` 3. **Range selectivity algorithm**: For predicates like `col > 100 AND col < 200`: - Groups multiple comparisons on same column to analyze as unified range constraint - Tracks lower and upper bounds, IN lists, and equality constraints - Computes intersection of constraint range with column's min/max range - For IN clauses, prunes IN list to intersection and uses list size / cardinality - For numeric types: selectivity = (effectiveMax - effectiveMin) / (colMax - colMin) - For VARCHAR: uses first character ASCII value for range approximation - For other types: conservative 0.1 default selectivity 4. **Constraint propagation**: When `updateConstraints=true`, equality predicates update a constraint map: - For `a = b`, creates intersection constraint with min(distinct_in_overlap), stores for both column IDs - Subsequent predicates on same columns use updated constraints for more accurate estimates - Enables correlation-aware selectivity estimation across multiple filters 5. **StatisticsBuilder.cpp fixes**: Critical bug fix for min/max variant creation: - IntegerStatisticsBuilder uses int64_t internally but must create Variants with correct TypeKind - Added switch statement to cast int64_t to proper type (TINYINT→int8_t, SMALLINT→int16_t, INTEGER→int32_t, BIGINT→int64_t, HUGEINT→int128_t) - Similar fix for DoubleStatisticsBuilder: REAL columns need float Variants, not double - Without this fix, comparison selectivity would fail due to type mismatches between Variants 6. **Integration changes**: - Moved `flattenAll()` from anonymous namespace to public in DerivedTable.cpp (needed by Filters.cpp to flatten AND/OR expressions) - Added helper functions in PlanUtils.h for working with expression trees - Updated Schema.cpp/Schema.h to support constraint tracking - Modified ToGraph.cpp to use filter selectivity for cardinality estimation - Updated RelationOp to propagate constraints through plan nodes 7. **FiltersTest.cpp (900+ lines)**: Comprehensive test suite covering: - Selectivity validation (trueFraction and nullFraction in valid ranges) - Conjunction and disjunction combining with various NULL fractions - Comparison selectivity for all operators (=, <, <=, >, >=) with different range configurations - Range selectivity with multiple predicates on same column - IN clause selectivity with list pruning and intersection - Constraint propagation through equality predicates - Edge cases: empty ranges, no overlap, full overlap, NULL handling - Type-specific tests for INTEGER, BIGINT, REAL, DOUBLE, VARCHAR, TINYINT, SMALLINT 8. **Mathematical soundness**: - All selectivity calculations preserve proper three-valued logic (TRUE/FALSE/NULL) - `validateSelectivity()` enforces: 0 ≤ trueFraction ≤ 1, 0 ≤ nullFraction ≤ 1, trueFraction + nullFraction ≤ 1 - NULL semantics: NULL AND TRUE = NULL, NULL OR FALSE = NULL, comparisons with NULL = NULL - Smooth interpolation near minimum selectivity bounds to avoid optimizer instability The motivation for this work is to enable cost-based optimization decisions that depend on accurate cardinality estimates after filter application. This is particularly critical for: - Join ordering: choosing to filter high-selectivity predicates early - Hash join vs. merge join selection based on filtered cardinality - Partition count selection for hash aggregations - Memory budget allocation for operators The implementation is designed to be extensible for future enhancements like: - Histogram-based selectivity estimation - Multi-column correlation analysis - Machine learning-based selectivity models - Query feedback loop for selectivity correction Differential Revision: D88164653
oerling
force-pushed
the
export-D89130357
branch
from
82e5bf5 to
e953931
Compare
) Summary: Pull Request resolved: facebookincubator#708 This diff implements end-to-end constraint propagation through the Axiom query plan construction process. Building on the filter selectivity estimation framework (D88164653), this change tracks refined Value constraints (min/max ranges, cardinalities, null fractions) from leaf scans through joins, filters, projects, and aggregations, enabling more accurate cardinality estimates at every level of the plan. The core motivation is to improve cost-based optimization decisions by maintaining up-to-date Value metadata as constraints are derived from predicates and joins. For example, after a filter `WHERE age > 18 AND age < 65`, the optimizer now knows the age column has min=18, max=65, and reduced cardinality, which influences downstream join cost estimates. Similarly, join equalities like `customer.id = order.customer_id` update both columns' constraints to reflect the intersection of their ranges and cardinalities. This constraint tracking is essential for: - Accurate join cardinality estimation (especially for multi-way joins) - Proper null handling in outer joins (tracking which columns become nullable) - Aggregate cardinality estimation (result size of GROUP BY) - Project cardinality propagation (computing distinct counts through expressions) - Filter selectivity estimation (more accurate with refined input constraints) Main changes: 1. **PlanState constraint tracking** (Plan.h, Plan.cpp): - Added `ConstraintMap constraints` field to PlanState to track Value constraints by expression ID during plan construction - Added `exprConstraint()` function that derives constraints for expressions based on their type: - Literals: use the literal's min/max (already set in expr->value()) - Columns: look up in state.constraints, fall back to expr->value() - Field access: propagate cardinality from base expression - Aggregates: set cardinality from state.cost.cardinality (group count) - Function calls: use functionConstraint metadata if available, otherwise max of argument cardinalities - Modified Plan constructor to populate Plan::constraints from state.constraints using QueryGraphContext::registerAny() for lifetime management - Modified NextJoin to preserve constraints across join candidates - Added PlanStateSaver to save/restore constraints along with cost and placed sets 2. **Join constraint propagation** (RelationOp.cpp): - Added `addJoinConstraint()` function to update constraints for join key pairs: - For inner joins: both sides become non-nullable (nullFraction=0), ranges intersect via columnComparisonSelectivity - For outer joins: optional side becomes nullable, nullFraction set to (1 - innerFanout) - Non-key columns from optional side get nullFraction = (1 - innerFanout) - Added `addJoinConstraints()` to apply constraint updates for all key pairs - Modified Join constructor to: - Accept new `innerFanout` parameter (fanout if this were an inner join, used for null fraction calculation) - Call addJoinConstraints() with join type information - Handle filter expressions using conjunctsSelectivity() to get filter selectivity - For semi-joins and anti-joins, multiply fanout by filter selectivity - For semi-project (mark joins), update the mark column's trueFraction - Propagate non-key nullable constraints for outer joins 3. **Filter constraint integration** (Filters.cpp, Filters.h): - Modified `value()` function to first check state.constraints before falling back to expr->value() - Added `addConstraint()` helper that enforces type-based cardinality limits: - BOOLEAN: max 2 distinct values - TINYINT: max 256 distinct values - SMALLINT: max 65,536 distinct values - Modified `conjunctsSelectivity()` to call `exprConstraint()` for each conjunct before computing selectivity, ensuring constraints are computed and cached - Changed `exprSelectivity()` and `conjunctsSelectivity()` signatures to take non-const `PlanState&` to allow updating state.constraints - Added `constraintsString()` debugging helper to format ConstraintMap as readable string 4. **Project constraint propagation** (RelationOp.cpp): - Modified Project constructor to accept PlanState parameter - Added loop to derive and store constraints for each output column by calling exprConstraint() on projection expressions - This ensures that projected expressions (e.g., `col + 1`, `upper(name)`) have accurate cardinality estimates 5. **Aggregation constraint propagation** (RelationOp.cpp): - Modified Aggregation constructor to accept PlanState parameter - Modified `setCostWithGroups()` to accept PlanState - Grouping keys get cardinality from the aggregation's result cardinality (number of groups) - Aggregate functions get cardinality from exprConstraint() evaluation 6. **VeloxHistory integration** (VeloxHistory.cpp, VeloxHistory.h): - Added `setBaseTableValues()` function to update BaseTable column Values from ConstraintMap - Modified `findLeafSelectivity()` to always: 1. Call conjunctsSelectivity() with updateConstraints=true to get constraints 2. Update BaseTable column Values using setBaseTableValues() 3. Optionally sample if sampling is enabled - This ensures filter-derived constraints (min/max, cardinality, null fractions) are applied to base table columns before planning 7. **Optimization.cpp updates**: - Updated all RelationOp construction sites to pass PlanState: - Project: added state parameter to constructor calls - Aggregation: added state parameter, including planSingleAggregation() - Join: added innerFanout parameter and state - Filter: already had state - Modified PrecomputeProjection::maybeProject() to accept PlanState parameter - Updated makeDistinct() to accept PlanState - Updated Join::makeCrossJoin() to accept PlanState - Ensured PlanStateSaver preserves constraints when exploring alternative join orders 8. **QueryGraphContext lifetime management** (QueryGraphContext.h): - Added `registerAny()` template function to take ownership of arbitrary objects (stored as shared_ptr<void>) - Added `ownedObjects_` set and `mutex_` for thread-safe lifetime management - Used to manage ConstraintMap pointers in Plan objects, ensuring they remain valid throughout optimization 9. **Schema.h/Schema.cpp updates**: - Changed Value::cardinality from const to non-const to allow constraint updates - Added Value assignment operator that validates type equality before assigning - Added Value::toString() for debugging 10. **ToGraph.cpp updates**: - Updated all plan construction to pass PlanState to constructors - Modified constant deduplication to set min/max for literals to the literal value 11. **FunctionRegistry.h extension**: - Added optional `functionConstraint` callback to FunctionMetadata - Allows functions to provide custom constraint derivation logic - Returns `std::optional<Value>` with refined constraints for the function result 12. **Comprehensive test coverage** (ConstraintsTest.cpp, 347 lines): - `scanEquality`: Tests join equality constraints (n_nationkey = r_regionkey), verifies min/max ranges intersect and cardinality is limited to intersection size - `aggregateConstraint`: Tests grouping key cardinality propagates to all output columns - `projectConstraint`: Tests projected expression cardinality (col+1, col+col) inherits from source columns - `outer`: Tests outer join null fraction propagation to optional side (left join, right side becomes nullable with nullFraction ≈ 0.8) - `bitwiseAnd`: Tests custom functionConstraint for bitwise_and, verifies min=0 and max=min(arg1.max, arg2.max) The implementation maintains the invariant that state.constraints contains the most up-to-date Value metadata for all expressions in the current plan state. When exploring alternative join orders or plan structures, PlanStateSaver ensures constraints are properly saved and restored. This enables accurate "what-if" analysis during optimization without polluting the global expression Value metadata. The constraint propagation integrates seamlessly with the filter selectivity estimation from D88164653: - Filters call conjunctsSelectivity() which updates state.constraints - Joins use columnComparisonSelectivity() which updates constraints for join keys - All downstream operators see refined constraints via value(state, expr) - Cost estimation at each operator uses the most accurate available cardinality Future enhancements enabled by this infrastructure: - Constraint-based partition pruning (skip partitions outside min/max range) - Dynamic filter pushdown (propagate join-derived constraints to scans) - Constraint-based empty result detection (min > max indicates zero rows) - Correlation detection (tracking when columns have matching values) Differential Revision: D89130357
oerling
force-pushed
the
export-D89130357
branch
from
e953931 to
8f5c82b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Differential Revision: D89130357