Do not cache empty query results in QueryCache - #735
Open
olowosulu wants to merge 3 commits into
Open
Conversation
QueryCache.Get() and QueryCache.GetUnfiltered() unconditionally write the fetched entry into their respective maps, even when the result set is empty and no error occurred. Because the cache is package-scoped and lives for the entire test binary run, a single transient empty response poisons every subsequent lookup for that metric. Later tests that call Get() for the same metric receive the cached empty slice instead of re-querying the backend. Guard both cache writes with: if len(entry.results) > 0 || entry.err != nil This ensures that errors are still cached (preserving existing behaviour) while empty successful results are not stored. Behavioural note: without a retry loop, declining to cache an empty result means that each subsequent call for a genuinely absent metric will re-query the backend rather than returning instantly from the cache. This increases query volume for metrics that truly do not exist, but it is the correct trade-off because transient absence is indistinguishable from permanent absence at query time, and poisoning the cache is the more severe failure mode.
musa-asad
reviewed
Aug 6, 2026
musa-asad
approved these changes
Aug 6, 2026
- Add comma-ok checks at waiter map read sites so absent entries (empty result not cached) return nil, nil for retry - Add query_cache_test.go with 7 unit tests covering the cache predicate: empty-not-stored, non-empty-stored, error-stored, waiter-gets-nil-on-empty, unfiltered variants, and promqlMetricSelector
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
QueryCachecaches empty successful results, which poisons every later lookup for thesame metric within a test binary run.
In
Get()andGetUnfiltered()the cache write executed unconditionally:When a query legitimately returns zero results with no error, that empty
cacheEntryis stored for the remainder of the run. Every subsequent call for that metric then hits
the cache and receives the empty result instead of re-querying, even though the metric
may have become available in the meantime.
Blast radius
The cache is package-level state created once per test package in each
setup_test.go.The impact is therefore largest in
test/otel/standard/, where more than ten test filesshare a single
queryCache. A single early transient miss on a metric can turn intofailures across every later test in that package that queries the same metric.
The effect is confined to one test binary. Each
test/otel/*package builds its owncache, so this does not propagate between packages.
Fix
Cache only when there is something worth caching:
Errors are still cached, matching the previous behaviour. Only empty successful results
are left uncached.
Behavioural notes for reviewers
Two consequences worth being explicit about.
First, declining to cache an empty result means each subsequent call re-queries the
backend for that metric. This is the intended behaviour, but it does increase query
volume for a metric that is genuinely absent, where previously exactly one query was
made. In practice a genuinely absent metric fails the assertion on the first call, so
the additional volume is bounded in the common case.
Second, the concurrent waiter path in
Get()now has a newly reachable state. Waitersthat block on the inflight channel read the cache afterwards without an existence check:
Previously an entry was always present. Now the key can be absent, in which case Go
returns the zero
cacheEntryand the waiter returns(nil, nil). That is exactly whatthe fetching goroutine returns for the same query, so behaviour is consistent and
correct. It is worth noting that this relies on the zero value of
cacheEntrybeingsemantically equivalent to an empty result, so a future field whose zero value carries
meaning would need this path revisited. I left the diff minimal rather than adding
explicit
, okchecks at the three read sites, but I am happy to add them if you wouldprefer the invariant made explicit.
The locking and inflight dedup design is otherwise unchanged.
Verification
Verified with Go 1.25.8, matching the
~1.23.0used in CI rather than thego 1.20ingo.mod, which is stale relative to the code (util/otelmetrics/query_cache.goalreadyimports
log/slog, which requires Go 1.21+).go build ./util/otelmetrics/passesgo vet ./util/otelmetrics/passesgo test -c -tags integrationcompiles for./test/otel/standard/,./test/otel/neuron/and
./test/otel/multi_efa/These are integration tests requiring live AWS infrastructure, so I was able to verify
compilation but not execution.
Scope
This change is deliberately limited to the caching bug so it can be reviewed on its own
merits. It is independent of any test-flakiness question and is provable from the code
alone.