Always send an explicit LIMIT on BanyanDB queries - #13961
Merged
Conversation
BanyanDB applies its own default limit to any query that carries none -- 100 rows for measures, 20 for streams/traces -- and applies it after GROUP BY, so an over-long result set is silently truncated rather than rejected. OAP never sent a limit on several read paths, so an entity-scoped metrics query returned at most 100 data points regardless of the requested range: a 4-hour minute-step read rendered only its first 100 minutes and the rest showed as empty, even though DurationUtils.MAX_TIME_RANGE allows up to 500 steps. The same cap silently shortened topology relation maps, instance and process metadata lists, profiling thread snapshots and eBPF task lists. Every BydbQL query now leaves OAP with an explicit LIMIT: - the entity-scoped metrics read sends the exact number of assembled duration points, the same row set the ES/JDBC DAOs fetch by explicit id; - ad-hoc SELECT TOP sends its own N; - anything that does not paginate itself falls back to the configured resultWindowMaxSize (default 10000) via Conditions#limitIfAbsent, applied in the stream/measure/trace query helpers that every DAO funnels through. The fallback is spliced in at the start of the pagination tail rather than appended, so it lands ahead of an OFFSET that was set first and keeps WITH QUERY_TRACE positioned as the grammar requires. ES and JDBC storage were never affected -- both fetch metrics rows by explicit document id.
hanahmily
approved these changes
Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix silent truncation of BanyanDB query results at the engine's implicit row cap — #13960
Why the bug exists. BanyanDB applies its own default limit to any query that carries none —
defaultLimit uint32 = 100for measures (pkg/query/logical/measure/measure_analyzer.go, mirrored in the vectorized planner) and20for streams/traces. The analyzer applies it after anyGROUP BY(plan = limit(plan, offset, limitParameter)wraps the whole plan), so an over-long result set is silently truncated rather than rejected.OAP never sent a
LIMITon several read paths.BanyanDBMetricsQueryDAO.queryByEntityIDbuiltConditions.create().eq(ENTITY_ID, id)with no limit, so an entity-scoped metrics query returned at most 100 data points regardless of the requested range — a 4-hour minute-step read (240 points) rendered only its first 100 minutes, and the remaining 140 fell into theemptyValue = truebranch. That is well inside what the query layer considers legal:DurationUtils.MAX_TIME_RANGE = 500, which throws a descriptive error rather than truncating.The same cap silently shortened topology relation maps (6 limit-less queries, including the whole-range ones), instance/process metadata lists, profiling thread snapshots, eBPF task lists, and the
multiGetread-before-write in the metrics aggregation path.This is not a regression from the BydbQL migration (#13947) —
git show 5f6496e72c^has the same limit-less query, anddefaultLimit = 100has been in BanyanDB since apache/skywalking-banyandb#234.How it is fixed. Every BydbQL statement now leaves OAP with an explicit
LIMIT, so nothing depends on the engine's default:readMetricsValues/readLabeledMetricsValues/readHeatMap) sendsLIMIT = duration.assembleDurationPoints().size()— the exact row set the ES and JDBC DAOs fetch by explicit document id, so the only remaining ceiling is the existingDurationUtils.MAX_TIME_RANGEgate. Ad-hocSELECT TOPsends its ownN. Paginated reads (traces, logs, alarms, events, browser logs) already passed the caller'slimit/fromand are unchanged.resultWindowMaxSizeknob (default10000,SW_STORAGE_BANYANDB_QUERY_MAX_WINDOW_SIZE) via a newConditions#limitIfAbsent, applied in the stream / measure / trace query helpers inAbstractBanyanDBDAOthat every DAO funnels through. The knob's documented meaning is widened accordingly inbydb.ymland the storage doc. It is read offBanyanDBStorageClientso the config does not have to be threaded through ~30 DAO constructors.The fallback is spliced in at the start of the pagination tail rather than appended, so it still lands ahead of an
OFFSETthat was set first (... LIMIT ? OFFSET ?) and keepsWITH QUERY_TRACEat the position the grammar mandates. It is a no-op when the caller set a limit, and idempotent.SHOW TOPneeds no change — it maps to aTopNRequest, which has noLIMITclause in the grammar and does not go through the measure analyzer's limit path.Behaviour note for reviewers. These paths now return more rows than before — that is the fix, but a very large deployment may see bigger topology / metadata payloads than it used to, where the previous size was only "correct-looking" because it had been cut at 100. Unlike ES's
index.max_result_window, BanyanDB does not reject an over-large window, soresultWindowMaxSizeis the operative bound on those reads.Tests.
ConditionsTestgains 8 cases coveringlimitIfAbsent(appends when absent, keeps a caller limit, idempotent, splices ahead of an existingOFFSETwith parameters re-aligned, composes afterGROUP BY/ORDER BY, and keepsWITH QUERY_TRACEahead of pagination in both layouts). NewQueryLimitTestpins the invariant at the DAO funnel — that a measure query with no caller limit, with no conditions at all, and with aGROUP BYall leave carrying aLIMIT, and that an explicit caller limit is not doubled. 41 tests pass in the module.Storage scope. ES and JDBC were never affected:
MetricsQueryEsDAOissues a multi-get over the exact document ids andJDBCMetricsQueryDAOusesid IN (...)over the assembled points, so neither has a size cap on this path.CHANGESlog.