Forbid Batch Sorted Merge on multikey minmax index - #9955
Merged
Conversation
|
@antekresic, @svenklemm: please review this pull request.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
natalya-aksman
added
Columnstore
natalya-aksman
force-pushed
the
forbid_batch_sorted_merge_on_multikey_minmax
branch
from
June 2, 2026 21:18
6ca04a9 to
a5a4bca
Compare
antekresic
reviewed
Jun 3, 2026
dbeck
approved these changes
Jun 3, 2026
Poroma-Banerjee
approved these changes
Jun 3, 2026
natalya-aksman
force-pushed
the
forbid_batch_sorted_merge_on_multikey_minmax
branch
from
June 3, 2026 16:22
a5a4bca to
770fff8
Compare
akuzm
reviewed
Jun 4, 2026
| * Batches can be sorted incorrectly on multikey minmax index, | ||
| * for example | ||
| * [(1, 20) .. (1, 30), (2,0)...(2,30)] with min(1),(0) | ||
| * will be sorted before [(1,1) .. (1,19)] with min(1),(1) |
Member
There was a problem hiding this comment.
Shouldn't we just continue adding the batches to the heap if there's a tie on leading columns? Disabling it altogether looks too heavy-handed.
Member
Author
There was a problem hiding this comment.
It is only disabled if there are no firstlast indexes on the non-leading columns. No firstlast on leading column is OK, BSM is not disabled in this case, there is a unit test for it.
Shouldn't we just continue adding the batches to the heap if there's a tie on leading columns?
Batch Sorted Merge depends on batches added in correct sort order of their first tuple, adding [(1, 20) .. (1, 30), (2,0)...(2,30)] before [(1,1) .. (1,19)] when we sort on 2 keys may lead to wrong result order as in the issue reproducer.
Merged
surister
pushed a commit
that referenced
this pull request
Jun 16, 2026
## 2.28.0 (2026-06-16) This release contains performance improvements and bug fixes since the 2.27.2 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.28.0** * **Faster `first()` and `last()` queries on compressed data.** TimescaleDB derives `first(value, time)` and `last(value, time)` aggregates straight from the columnstore's batch metadata, skipping batch decompression entirely. For the "latest reading per series" lookups that time-series workloads run constantly, that means meaningfully faster recency queries with no changes to your SQL queries. * **Lighter, less disruptive continuous aggregate refreshes.** `refresh_continuous_aggregate()` can now run incrementally in batches — the same behavior refresh policies already use — enabling breaking large manual refreshes into smaller chunks (tunable via `buckets_per_batch`, `max_batches_per_execution`, and `refresh_newest_first`) instead of one heavy operation. Refreshes also now take a lighter lock while processing the invalidation log, so they no longer block unrelated concurrent operations on the same continuous aggregate, improving behavior for concurrent workloads. * **Vectorized execution now covers `CASE` expressions.** TimescaleDB's columnar executor can now evaluate `CASE ... WHEN` expressions directly on compressed data, so queries using conditional logic stay on the fast vectorized path instead of falling back to slower row-by-row decompression. This speeds up a common pattern — conditional aggregations and computed columns over compressed history — with no query changes needed. * **Add new aggregations to a continuous aggregate without rebuilding it.** You can now run `ALTER MATERIALIZED VIEW <cagg> ADD COLUMN <name> <type> GENERATED ALWAYS AS (<aggregate>) STORED` to add a new computed aggregate to an existing continuous aggregate in place — no more dropping and recreating the whole aggregate just to track one more metric. New data populates the column going forward, letting your rollups evolve alongside your application. (Existing rows start as `NULL`; a forced refresh backfills them when you need historical values.) **Deprecation Notice: PostgreSQL 15 Support** This release marks the final minor version of TimescaleDB that will support PostgreSQL 15. Starting with our next release, version 2.29.0, we will officially drop support for Postgres 15, and only support Postgres 16, 17, and 18; however, all future patch releases within the current 2.28 version cycle will continue to fully support it. We recommend planning your PostgreSQL upgrades accordingly to ensure a smooth transition. **Deprecation Notice: `chunk_constraint` Catalog Table** Please note that the `_timescaledb_catalog.chunk_constraint` table has been dropped and temporarily replaced by a view, which introduces a change to the underlying objects while maintaining current query behavior. However, this compatibility view will be completely removed in a future release. To ensure your queries remain compatible moving forward, we strongly advise transitioning to the stable contracts provided by our [informational views](https://www.tigerdata.com/docs/reference/timescaledb/informational-views). **Backward-Incompatible Changes** * [#9934](#9934) Remove adaptive chunking **Features** * [#4054](#4054) Support `ANALYZE` and `VACUUM` on continuous aggregates by redirecting to the underlying materialization hypertable * [#9125](#9125) Increase the parallelism of `SELECT` queries over compressed hypertables to approximately match the uncompressed data size * [#9410](#9410) Mark `hypertable` and `chunk` as user catalog tables * [#9416](#9416) Support some forms of `CASE` expression in columnar aggregation and grouping * [#9580](#9580) Add `first` / `last` sparse indexes to compression * [#9784](#9784) Use `first` / `last` sparse index for `orderby` metadata on new compressed chunks * [#9668](#9668) Allow database owner to configure hypertables and policies * [#9701](#9701) Relax lock during continuous aggregate invalidation log processing * [#9730](#9730) Add in-memory observability for compressed chunks * [#9735](#9735) Improve `GapFill` row count estimate * [#9821](#9821) Allow subquery results which are exec params as GapFill arguments * [#9825](#9825) Support `ADD COLUMN` on continuous aggregates * [#9842](#9842) Suppress continuous aggregate invalidation tracking during bulk loads * [#9878](#9878) Remove `chunk_constraint` catalog tracking for foreign keys * [#9893](#9893) Remove `chunk_constraint` catalog tracking for non-dimensional constraints * [#9903](#9903) Incremental refresh for `refresh_continuous_aggregate()` * [#9915](#9915) Remove `_timescaledb_catalog.chunk_constraint` table * [#9938](#9938) Add `rebuild_sparse_index` function * [#9964](#9964) Add a function to lock OSM chunk's dimension slice * [#9980](#9980) Support `first/last(value, time)` in `ColumnarIndexScan` **Bugfixes** * [#9708](#9708) Guard time bucket parameter handling against bad input * [#9745](#9745) Check constraints when adding unique constraints to chunks * [#9890](#9890) Fix incremental refresh batch boundaries to align with variable-width buckets and start only where a chunk and an invalidation overlap * [#9914](#9914) Fix use-after-free in segmentwise recompression * [#9929](#9929) Fix background jobs being bumped in the queue forever and never running * [#9955](#9955) Fix wrong results when using Batch Sorted Merge with no first-last index on a non-leading order by column * [#9967](#9967) Block upgrade after downgrade with first/last indexes present * [#9976](#9976) Fix wrong results when comparing a date column to a `timestamptz` value * [#9977](#9977) Fix `COPY WHERE` into a hypertable with dropped columns * [#9981](#9981) Fix set-returning functions in the sort key of `ColumnarScan` * [#9982](#9982) Reject `ALTER TABLE ... INHERIT` when the parent is a hypertable * [#9984](#9984) Fix handling of `NOT VALID NOT NULL` constraint for query optimization * [#9986](#9986) Handle `MERGE WHEN NOT MATCHED BY SOURCE` on hypertables * [#9988](#9988) Fix `time_bucket_gapfill` function detection * [#10003](#10003) Block unsafe updates of unique columns on compressed chunks * [#10024](#10024) Fix `approximate_row_count` handling of Infinity * [#10025](#10025) Fix rename on compressed continuous aggregates * [#10026](#10026) Fix chunk skipping near `PG_INT64_MAX` **New Settings** * `skip_cagg_invalidation`: skip continuous aggregate invalidation tracking for DML and DDL in the current session/transaction. Off by default. * `stats_max_chunks`: set the per-database compressed chunk statistics cache capacity. Defaults to 1024 chunks; set to 0 to disable the feature. **Thanks** * @Fabian-2596 for suggesting more accurate GapFill row count estimate * @otjdiepluong for fixing spelling mistakes in timescaledb source code comments * @scimad and @Nosfistis for suggesting expanding coverage for gapfill arguments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9922
Forbids Batch Sorted Merge when there is no firstlast index on a non-leading order by column.