Do logical sparse index comparison - #9736
Merged
Merged
Conversation
|
@akuzm, @antekresic: please review this pull request.
|
Poroma-Banerjee
force-pushed
the
tscompress
branch
5 times, most recently
from
May 8, 2026 01:40
d3c4351 to
b3a8828
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
svenklemm
reviewed
May 8, 2026
svenklemm
approved these changes
May 8, 2026
dbeck
approved these changes
May 8, 2026
Poroma-Banerjee
force-pushed
the
tscompress
branch
from
May 8, 2026 14:10
b3a8828 to
80f7d41
Compare
Sparse index JSONB arrays can have objects in different order while being logically identical, e.g. after ALTER TABLE changes the compress_index declaration order. Replace ts_jsonb_equal with a new ts_sparse_index_equal that parses both sides and matches objects as an unordered set.
Poroma-Banerjee
force-pushed
the
tscompress
branch
from
May 8, 2026 14:23
80f7d41 to
efca0d0
Compare
This was referenced May 8, 2026
Merged
surister
added a commit
that referenced
this pull request
May 12, 2026
## 2.27.0 (2026-05-12) This release contains performance improvements and bug fixes since the 2.26.4 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.27.0** * The Hypercore engine now supports a vectorized implementation of filters by evaluating them inline through the standard Postgres function path. This expands the set of queries (including continuous aggregate refreshes) that can take the faster path through the columnstore, yielding speedups ranging from 30% up to 2x in benchmarks. * `UPDATE` and `DELETE` statements with equality predicates can now use bloom filters to skip decompressing batches whose compressed rows can't match. When multiple bloom filters apply, they are evaluated in decreasing order of column count (most selective first), and EXPLAIN now reports filtering activity via the new "Compressed batches filtered" and "Batches filtered after decompression counters". The query performance increases in some case up to 160 times. * `UPSERT` queries can now leverage bloom filters (including composite ones) to skip decompressing batches when the arbiter values are guaranteed not to be present, with the most-selective filter chosen automatically when multiple apply. EXPLAIN output adds new statistics — batches checked by bloom, batches pruned by bloom, batches without bloom, and bloom false positives — for visibility into pruning effectiveness. **Upcoming PostgreSQL 15 EOL announcement** As a reminder, the upcoming TimescaleDB release in June 2026 will officially be the last version with support for PostgreSQL 15. This deprecation was initially announced in the [v2.23.0 changelog](https://github.com/timescale/timescaledb/releases/tag/2.23.0) on October 29, 2025, to provide users ample time to prepare. To ensure uninterrupted access to new features, bugfixes and performance enhancements, all instances must be upgraded to PostgreSQL 16 or greater. **Backward-Incompatible Changes** * [#9579](#9579) The bloom filter sparse indexes on compressed `int2` columns could lead to `SELECT` queries not returning the rows that actually match the `WHERE` condition. The upgrade is blocked for the affected databases, and the incorrect indexes have to be dropped manually before the upgrade. * This release introduces a new naming convention for composite bloom filter metadata. While this change will not disrupt query processing, v2.27 cannot automatically utilize composite bloom filters generated in v2.26. To convert your existing v2.26 composite bloom filters, the legacy metadata columns must be renamed. This is a lightweight, catalog-only operation requiring zero data recompression, which can be done with [this migration script](https://github.com/timescale/timescaledb-extras/blob/main/utils/2.27.x-fix-composite-bloom-columns.sql). **Features** * [#8868](#8868) Use `PG_MODULE_MAGIC_EXT` for `PG18` * [#8967](#8967) Rewriting queries with continuous aggregates exactly matching query aggregation * [#9192](#9192) Push down scalar array operations into the columnar metadata scan by transforming them into an `OR`/`AND` clause. * [#9355](#9355) Defer `segmentby` default for direct compress * [#9374](#9374) Use bloom filters to eliminate decompression of unrelated compressed batches during `UPSERT`s. * [#9396](#9396) Analyze and get `segmentby` during direct compress * [#9398](#9398) Fix chunk exclusion for `IN`/`ANY` on open (time) dimensions * [#9399](#9399) Use bloom filters to reduce decompression during `UPDATE`/`DELETE` commands. * [#9403](#9403) Set default `segmentby` during direct compress flush * [#9437](#9437) Allow running compression as part of refresh policy for compressed continuous aggregates * [#9443](#9443) Enable vectorized aggregation in some cases when the `WHERE` clause contains filters not handled through the "Vectorized Filters" facility. This includes e.g. filters on `time_bucket()`. * [#9458](#9458) Remove `_timescaledb_functions.repair_relation_acls` * [#9475](#9475) Calculate hashes for bloom filter predicates at planning time. * [#9504](#9504) Allow `ALTER TABLE RESET` on materialization hypertables * [#9521](#9521) Add support for reporting index creation progress * [#9559](#9559) Notice on compression settings change * [#9569](#9569) For nullable `orderby` columns do segmentwise decompress-compress instead of segmentwise recompress. * [#9583](#9583) Drop existing sparse indexes when dropping columns * [#9648](#9648) Support `ENABLE`/`DISABLE TRIGGER` on hypertables * [#9702](#9702) Allow Batch Sorted Merge for unordered chunks with no `segmentby` or when all `segmentby` columns are pinned to a `Const` **Bugfixes** * [#9363](#9363) Change compression job status when chunks could be compressed * [#9413](#9413) Fix incorrect decompress markers on full batch delete * [#9414](#9414) Fix `NULL` compression handling in `estimate_uncompressed_size` * [#9417](#9417) Fix segfault in `bloom1_contains` * [#9479](#9479) Disallow sub-day offset for `time-bucket` on `Date` * [#9482](#9482) Forbid Batch Sort Merge on nullable `orderby` columns * [#9490](#9490) Disallow negative interval as `chunk_interval` * [#9500](#9500) Fix off-by-one error when building object name * [#9519](#9519) Remove self-referential `FOREIGN KEY` constraints from catalog * [#9561](#9561) Simplify job history retention by replacing binary search and temp table * [#9590](#9590) Fix policy skipping uncompressed chunks * [#9596](#9596) Remove unused `process_hypertable_invalidations` policy code * [#9604](#9604) Remove dead `post_parse_analyze_hook` capture in loader * [#9610](#9610) Fix use-after-free crash in `cache_destroy` during transaction abort * [#9632](#9632) Preserve chunk settings during recompress * [#9640](#9640) Fix `NULL` `datumCopy` crash in `segmentby` analysis * [#9680](#9680) Fix segfault in direct compress insert on hypertable with dropped column * [#9692](#9692) Fix internal "invalid perminfoindex 0 in RTE" error on `MERGE NOT MATCHED INSERT` into a hypertable * [#9705](#9705) Avoid double `TOAST` delete when `DELETE-after-compression` is enabled * [#9705](#9705) Only freeze compressed rows when truncating uncompressed chunk * [#9706](#9706) Use `bigint` in `estimate_uncompressed_size` calculations * [#9709](#9709) Reject mismatched element type in `bool`/`uuid` decompression * [#9710](#9710) Return `bigint` from `compressed_data_column_size` * [#9711](#9711) Fix registration row leak when continuous aggregate refresh fails * [#9697](#9697) Improve `pathkey` handling for compressed sub-paths during sort transformation * [#9743](#9743) Fix the composite bloom metadata column naming scheme * [#9767](#9767) Skip dropped chunks when trying to remove `ts_cagg_invalidation_trigger` * [#9747](#9747) Reject inheriting from a hypertable * [#9744](#9744) Use a fixed call string for the telemetry job in `ts_stat_statements` recording * [#9736](#9736) Do logical sparse index comparison * [#9731](#9731) Avoid creating overlapping batches during recompression for multi orderby configurations * [#9717](#9717) Reject non-positive time bucket width on cagg creation * [#9707](#9707) Fix policy name comparison in remove_policies **New Settings** * `enable_cagg_rewrites`: enables rewriting queries with CAggs. Off by default. `cagg_rewrites_debug_info`: prints CAgg rewrites diagnostics. Off by default. * `enable_columnar_scan_filter_pushdown`: enables pushing the filters on columnar scan down to the compressed scan level. On by default. **Thanks** * @fabriziomello for adding support for `PG_MODULE_MAGIC_EXT` * @maltalex for reporting an issue with index creation progress reporting * @pavanmanishd for the first version of the fix for #9743 * @h0rn3t for reporting issue with recompression creating overlapping batches --------- Co-authored-by: timescale-automation <123763385+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ivan Sanchez <surister98@gmail.com> Co-authored-by: philkra <philip@philipkrauss.at>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sparse index JSONB arrays can have objects in different order while
being logically identical, e.g. after ALTER TABLE changes the
compress_index declaration order. Replace ts_jsonb_equal with a new
ts_sparse_index_equal that parses both sides and matches objects as
an unordered set.