Open
Conversation
Contributor
|
💻 Deploy preview available (MQE: subset selector elimination): |
…r subset selectors
charleskorn
force-pushed
the
charleskorn/subset-selector-elimination
branch
from
45147bb to
a25485e
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Outdated
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
Contributor
Author
Return the actual error from introduceDuplicateNode instead of nil when an error occurs. This ensures that failures during duplicate node introduction are properly propagated instead of being silently ignored, which could lead to inconsistent query plan state. Applied via @cursor push command
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/instant_vector_operator.go
Show resolved
Hide resolved
This comment has been minimized.
This comment has been minimized.
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Show resolved
Hide resolved
|
Bugbot Autofix prepared fixes for 1 of the 1 bugs found in the latest run.
Or push these changes by commenting: Preview (1c3d753413)diff --git a/pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go b/pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
--- a/pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
+++ b/pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
@@ -710,8 +710,14 @@
return false, true
}
- return !group.haveAnyFiltersForLabel(destinationLabelName), true
+ if group.haveAnyFiltersForLabel(destinationLabelName) {
+ return false, true
+ }
+ // These functions drop the __name__ label if delayed name removal is not enabled, so it's only safe to apply
+ // filtering after these functions if delayed name removal is enabled, or there is no filtering by __name__.
+ return delayedNameRemovalEnabled || !group.haveAnyFiltersForLabel(model.MetricNameLabel), true
+
case functions.FUNCTION_HISTOGRAM_FRACTION, functions.FUNCTION_HISTOGRAM_QUANTILE:
// These functions drop the 'le' label on native histograms, so we can never apply filtering after the function
// if the filter is on that label. |
tacole02
reviewed
Feb 17, 2026
Contributor
tacole02
left a comment
tacole02
left a comment
There was a problem hiding this comment.
Docs look good! Thank you!
charleskorn
commented
Feb 17, 2026
56quarters
reviewed
Feb 18, 2026
Contributor
56quarters
left a comment
56quarters
left a comment
There was a problem hiding this comment.
I haven't finished going through this but it's EOD here.
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Show resolved
Hide resolved
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Show resolved
Hide resolved
pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.go
Show resolved
Hide resolved
# Conflicts: # cmd/mimir/help-all.txt.tmpl # pkg/streamingpromql/engine.go # pkg/streamingpromql/planning.go # pkg/streamingpromql/planning/plan.go # pkg/streamingpromql/planning/plan.pb.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
This PR adds support for subset selector elimination (SSE) in MQE, building on common subexpression elimination (CSE).
Where CSE eliminates repeated expressions like
aina / (a + b), SSE avoids reevaluating expressions that are a subset of another expression, such asrequest_count{status="success"}inrequest_count{status="success"} / request_count.It also attempts to push down operations that can still be correctly applied before any subset filter is applied. For example, in
rate(request_count{status="success"}[5m]) / rate(request_count[5m]), applying thestatus="success"filter to series and then calculating the rate produces the same result as calculating the rate on all series and then applying the filter - so the optimization pass tries to identify opportunities to apply the filtering as late as possible, to avoid duplicate work.This improves query performance and reduces querier resource consumption, and reduces the load on store-gateways and ingesters, reducing their resource consumption.
This is a large PR. There are four main areas of change:
DuplicateFilternodes to filter the result of aDuplicatenode based on the provided matchers:pkg/streamingpromql/optimize/plan/commonsubexpressionelimination/optimization_pass.gosum(foo{status="success"}) / sum(foo)), based on the output from the CSE / SSE optimization pass. This allows this common kind of query to avoid buffering anything:pkg/streamingpromql/optimize/plan/multiaggregation/optimization_pass.gopkg/streamingpromql/optimize/plan/commonsubexpressionelimination/instant_vector_operator.go,range_vector_operator.goandseries_data_ring_buffer.gopkg/streamingpromql/optimize/plan/multiaggregation/operator.goOur benchmarks show no measurable impact to queries eligible to CSE despite the changes in this PR, and queries eligible for SSE show substantial improvement in latency (30-40% for non-trivial cases) with no increase in peak memory consumption.
There is one exception: the cases where the selector retrieves a single series. In these cases, the query evaluates significantly faster but has nearly 2x the peak memory consumption as before. I believe this is due to the fact that the queries finish very quickly and don't read the full response from ingester, so the gRPC stream is drained and closed asynchronously. This can overlap with the next iteration of the benchmark and therefore the next request to the ingester, causing the increased memory consumption that we see.
Benchmark results
Which issue(s) this PR fixes or relates to
(none)
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]. If changelog entry is not needed, please add thechangelog-not-neededlabel to the PR.about-versioning.mdupdated with experimental features.Note
Medium Risk
Touches core query planning and streaming evaluation/buffering logic; while gated behind an experimental flag and plan versioning, mistakes could cause incorrect query results or memory/CPU regressions in eligible queries.
Overview
Adds experimental subset selector elimination (SSE) to the Mimir Query Engine, allowing queries like
foo{env="bar"} / footo evaluate the broader selector once and apply per-consumer filtering via new plan nodeDuplicateFilter.Extends the common subexpression elimination pass to detect duplicate vs subset selectors, track new labeled elimination metrics, and only apply SSE when supported by
QueryPlanV7and safe to push filtering past certain functions/aggregations. Updates duplication buffering/operators to support filtered consumers and avoid buffering series that no remaining consumer will read.Wires a new flag/config option
-querier.mimir-query-engine.enable-subset-selector-elimination(defaultfalse), updates docs/help/changelog, and expands benchmarks/tests to cover SSE behavior and annotation differences.Written by Cursor Bugbot for commit 536be67. This will update automatically on new commits. Configure here.