Add rebuild_sparse_index function - #9938
Merged
Merged
Conversation
|
@melihmutlu, @natalya-aksman: please review this pull request.
|
Poroma-Banerjee
force-pushed
the
tscompress
branch
7 times, most recently
from
May 31, 2026 20:59
0b53807 to
edfd859
Compare
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
svenklemm
reviewed
Jun 1, 2026
antekresic
reviewed
Jun 1, 2026
antekresic
left a comment
antekresic
left a comment
Member
There was a problem hiding this comment.
Took a first pass, will look at it some more.
Poroma-Banerjee
force-pushed
the
tscompress
branch
from
June 1, 2026 13:51
edfd859 to
28998af
Compare
svenklemm
reviewed
Jun 2, 2026
svenklemm
reviewed
Jun 2, 2026
antekresic
reviewed
Jun 2, 2026
Poroma-Banerjee
force-pushed
the
tscompress
branch
5 times, most recently
from
June 2, 2026 14:38
d1d5fdc to
c1f15fe
Compare
antekresic
approved these changes
Jun 3, 2026
Poroma-Banerjee
force-pushed
the
tscompress
branch
from
June 3, 2026 14:41
c1f15fe to
647ffdc
Compare
svenklemm
reviewed
Jun 3, 2026
| { | ||
| char *meta_name = | ||
| compressed_column_metadata_name_v2(minmax_prefixes[i], &colname, 1); | ||
| Assert(get_attnum(compressed_relid, meta_name) != InvalidAttrNumber); |
Member
There was a problem hiding this comment.
could this ever be false? this also seems to contradict the missing_ok
Member
Author
There was a problem hiding this comment.
It is a good check for debug. it means something has gone wrong internally or was changed, and this particular place was forgotten.
missing_ok is true as it will not error out in production
svenklemm
reviewed
Jun 3, 2026
svenklemm
reviewed
Jun 3, 2026
svenklemm
approved these changes
Jun 3, 2026
Rebuild sparse index metadata columns on compressed chunks in-place without full recompression. Adds _timescaledb_functions.rebuild_sparse_index(chunk, force) which diffs chunk vs hypertable sparse index settings, drops/adds metadata columns, then populates them by decompressing each batch through the existing RowDecompressor and BatchMetadataBuilder pipeline.
Poroma-Banerjee
force-pushed
the
tscompress
branch
from
June 3, 2026 20:20
647ffdc to
2d64725
Compare
Merged
surister
pushed a commit
that referenced
this pull request
Jun 16, 2026
## 2.28.0 (2026-06-16) This release contains performance improvements and bug fixes since the 2.27.2 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.28.0** * **Faster `first()` and `last()` queries on compressed data.** TimescaleDB derives `first(value, time)` and `last(value, time)` aggregates straight from the columnstore's batch metadata, skipping batch decompression entirely. For the "latest reading per series" lookups that time-series workloads run constantly, that means meaningfully faster recency queries with no changes to your SQL queries. * **Lighter, less disruptive continuous aggregate refreshes.** `refresh_continuous_aggregate()` can now run incrementally in batches — the same behavior refresh policies already use — enabling breaking large manual refreshes into smaller chunks (tunable via `buckets_per_batch`, `max_batches_per_execution`, and `refresh_newest_first`) instead of one heavy operation. Refreshes also now take a lighter lock while processing the invalidation log, so they no longer block unrelated concurrent operations on the same continuous aggregate, improving behavior for concurrent workloads. * **Vectorized execution now covers `CASE` expressions.** TimescaleDB's columnar executor can now evaluate `CASE ... WHEN` expressions directly on compressed data, so queries using conditional logic stay on the fast vectorized path instead of falling back to slower row-by-row decompression. This speeds up a common pattern — conditional aggregations and computed columns over compressed history — with no query changes needed. * **Add new aggregations to a continuous aggregate without rebuilding it.** You can now run `ALTER MATERIALIZED VIEW <cagg> ADD COLUMN <name> <type> GENERATED ALWAYS AS (<aggregate>) STORED` to add a new computed aggregate to an existing continuous aggregate in place — no more dropping and recreating the whole aggregate just to track one more metric. New data populates the column going forward, letting your rollups evolve alongside your application. (Existing rows start as `NULL`; a forced refresh backfills them when you need historical values.) **Deprecation Notice: PostgreSQL 15 Support** This release marks the final minor version of TimescaleDB that will support PostgreSQL 15. Starting with our next release, version 2.29.0, we will officially drop support for Postgres 15, and only support Postgres 16, 17, and 18; however, all future patch releases within the current 2.28 version cycle will continue to fully support it. We recommend planning your PostgreSQL upgrades accordingly to ensure a smooth transition. **Deprecation Notice: `chunk_constraint` Catalog Table** Please note that the `_timescaledb_catalog.chunk_constraint` table has been dropped and temporarily replaced by a view, which introduces a change to the underlying objects while maintaining current query behavior. However, this compatibility view will be completely removed in a future release. To ensure your queries remain compatible moving forward, we strongly advise transitioning to the stable contracts provided by our [informational views](https://www.tigerdata.com/docs/reference/timescaledb/informational-views). **Backward-Incompatible Changes** * [#9934](#9934) Remove adaptive chunking **Features** * [#4054](#4054) Support `ANALYZE` and `VACUUM` on continuous aggregates by redirecting to the underlying materialization hypertable * [#9125](#9125) Increase the parallelism of `SELECT` queries over compressed hypertables to approximately match the uncompressed data size * [#9410](#9410) Mark `hypertable` and `chunk` as user catalog tables * [#9416](#9416) Support some forms of `CASE` expression in columnar aggregation and grouping * [#9580](#9580) Add `first` / `last` sparse indexes to compression * [#9784](#9784) Use `first` / `last` sparse index for `orderby` metadata on new compressed chunks * [#9668](#9668) Allow database owner to configure hypertables and policies * [#9701](#9701) Relax lock during continuous aggregate invalidation log processing * [#9730](#9730) Add in-memory observability for compressed chunks * [#9735](#9735) Improve `GapFill` row count estimate * [#9821](#9821) Allow subquery results which are exec params as GapFill arguments * [#9825](#9825) Support `ADD COLUMN` on continuous aggregates * [#9842](#9842) Suppress continuous aggregate invalidation tracking during bulk loads * [#9878](#9878) Remove `chunk_constraint` catalog tracking for foreign keys * [#9893](#9893) Remove `chunk_constraint` catalog tracking for non-dimensional constraints * [#9903](#9903) Incremental refresh for `refresh_continuous_aggregate()` * [#9915](#9915) Remove `_timescaledb_catalog.chunk_constraint` table * [#9938](#9938) Add `rebuild_sparse_index` function * [#9964](#9964) Add a function to lock OSM chunk's dimension slice * [#9980](#9980) Support `first/last(value, time)` in `ColumnarIndexScan` **Bugfixes** * [#9708](#9708) Guard time bucket parameter handling against bad input * [#9745](#9745) Check constraints when adding unique constraints to chunks * [#9890](#9890) Fix incremental refresh batch boundaries to align with variable-width buckets and start only where a chunk and an invalidation overlap * [#9914](#9914) Fix use-after-free in segmentwise recompression * [#9929](#9929) Fix background jobs being bumped in the queue forever and never running * [#9955](#9955) Fix wrong results when using Batch Sorted Merge with no first-last index on a non-leading order by column * [#9967](#9967) Block upgrade after downgrade with first/last indexes present * [#9976](#9976) Fix wrong results when comparing a date column to a `timestamptz` value * [#9977](#9977) Fix `COPY WHERE` into a hypertable with dropped columns * [#9981](#9981) Fix set-returning functions in the sort key of `ColumnarScan` * [#9982](#9982) Reject `ALTER TABLE ... INHERIT` when the parent is a hypertable * [#9984](#9984) Fix handling of `NOT VALID NOT NULL` constraint for query optimization * [#9986](#9986) Handle `MERGE WHEN NOT MATCHED BY SOURCE` on hypertables * [#9988](#9988) Fix `time_bucket_gapfill` function detection * [#10003](#10003) Block unsafe updates of unique columns on compressed chunks * [#10024](#10024) Fix `approximate_row_count` handling of Infinity * [#10025](#10025) Fix rename on compressed continuous aggregates * [#10026](#10026) Fix chunk skipping near `PG_INT64_MAX` **New Settings** * `skip_cagg_invalidation`: skip continuous aggregate invalidation tracking for DML and DDL in the current session/transaction. Off by default. * `stats_max_chunks`: set the per-database compressed chunk statistics cache capacity. Defaults to 1024 chunks; set to 0 to disable the feature. **Thanks** * @Fabian-2596 for suggesting more accurate GapFill row count estimate * @otjdiepluong for fixing spelling mistakes in timescaledb source code comments * @scimad and @Nosfistis for suggesting expanding coverage for gapfill arguments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuild sparse index metadata columns on compressed chunks in-place without full recompression.
Adds _timescaledb_functions.rebuild_sparse_index(chunk, force) which diffs chunk vs hypertable sparse index settings, drops/adds metadata columns, then populates them by decompressing each batch through the existing RowDecompressor and BatchMetadataBuilder pipeline.