Incremental refresh for refresh_continuous_aggregate() by melihmutlu · Pull Request #9903 · timescale/timescaledb

melihmutlu · 2026-05-26T13:22:16Z

refresh_continuous_aggregate() can now refresh incrementally in batches,
matching the behavior the continuous aggregate policy already supports.
Forced refresh is also incremental by default.

Options available for incremental refresh with policies can be set for
manual refresh via JSONB options:

buckets_per_batch (default 10, as policies)
max_batches_per_execution (default 0 = no limit)
refresh_newest_first

Usage:

Without options, it behaves similar to a refresh policy:

  CALL refresh_continuous_aggregate('cagg', t1, t2);

codecov · 2026-05-26T14:24:05Z

Codecov Report

❌ Patch coverage is 91.93548% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tsl/src/continuous_aggs/refresh.c	90.74%	6 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

github-actions · 2026-05-26T14:55:24Z

@pnthao, @Poroma-Banerjee: please review this pull request.

Powered by pull-review

kpan2034

One test needs a small fix, other than that LGTM

refresh_continuous_aggregate() can now refresh incrementally in batches, matching the behavior the continuous aggregate policy already supports. Forced refresh is also incremental by default. Options available for incremental refresh with policies can be set for manual refresh via JSONB options: - `buckets_per_batch` (default 10, as policies) - `max_batches_per_execution` (default 0 = no limit) - `refresh_newest_first` Usage: Without options, it behaves similar to a refresh policy with default values: CALL refresh_continuous_aggregate('cagg', t1, t2);

@Fabian-2596

## 2.28.0 (2026-06-16) This release contains performance improvements and bug fixes since the 2.27.2 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.28.0** * **Faster `first()` and `last()` queries on compressed data.** TimescaleDB derives `first(value, time)` and `last(value, time)` aggregates straight from the columnstore's batch metadata, skipping batch decompression entirely. For the "latest reading per series" lookups that time-series workloads run constantly, that means meaningfully faster recency queries with no changes to your SQL queries. * **Lighter, less disruptive continuous aggregate refreshes.** `refresh_continuous_aggregate()` can now run incrementally in batches — the same behavior refresh policies already use — enabling breaking large manual refreshes into smaller chunks (tunable via `buckets_per_batch`, `max_batches_per_execution`, and `refresh_newest_first`) instead of one heavy operation. Refreshes also now take a lighter lock while processing the invalidation log, so they no longer block unrelated concurrent operations on the same continuous aggregate, improving behavior for concurrent workloads. * **Vectorized execution now covers `CASE` expressions.** TimescaleDB's columnar executor can now evaluate `CASE ... WHEN` expressions directly on compressed data, so queries using conditional logic stay on the fast vectorized path instead of falling back to slower row-by-row decompression. This speeds up a common pattern — conditional aggregations and computed columns over compressed history — with no query changes needed. * **Add new aggregations to a continuous aggregate without rebuilding it.** You can now run `ALTER MATERIALIZED VIEW <cagg> ADD COLUMN <name> <type> GENERATED ALWAYS AS (<aggregate>) STORED` to add a new computed aggregate to an existing continuous aggregate in place — no more dropping and recreating the whole aggregate just to track one more metric. New data populates the column going forward, letting your rollups evolve alongside your application. (Existing rows start as `NULL`; a forced refresh backfills them when you need historical values.) **Deprecation Notice: PostgreSQL 15 Support** This release marks the final minor version of TimescaleDB that will support PostgreSQL 15. Starting with our next release, version 2.29.0, we will officially drop support for Postgres 15, and only support Postgres 16, 17, and 18; however, all future patch releases within the current 2.28 version cycle will continue to fully support it. We recommend planning your PostgreSQL upgrades accordingly to ensure a smooth transition. **Deprecation Notice: `chunk_constraint` Catalog Table** Please note that the `_timescaledb_catalog.chunk_constraint` table has been dropped and temporarily replaced by a view, which introduces a change to the underlying objects while maintaining current query behavior. However, this compatibility view will be completely removed in a future release. To ensure your queries remain compatible moving forward, we strongly advise transitioning to the stable contracts provided by our [informational views](https://www.tigerdata.com/docs/reference/timescaledb/informational-views). **Backward-Incompatible Changes** * [#9934](#9934) Remove adaptive chunking **Features** * [#4054](#4054) Support `ANALYZE` and `VACUUM` on continuous aggregates by redirecting to the underlying materialization hypertable * [#9125](#9125) Increase the parallelism of `SELECT` queries over compressed hypertables to approximately match the uncompressed data size * [#9410](#9410) Mark `hypertable` and `chunk` as user catalog tables * [#9416](#9416) Support some forms of `CASE` expression in columnar aggregation and grouping * [#9580](#9580) Add `first` / `last` sparse indexes to compression * [#9784](#9784) Use `first` / `last` sparse index for `orderby` metadata on new compressed chunks * [#9668](#9668) Allow database owner to configure hypertables and policies * [#9701](#9701) Relax lock during continuous aggregate invalidation log processing * [#9730](#9730) Add in-memory observability for compressed chunks * [#9735](#9735) Improve `GapFill` row count estimate * [#9821](#9821) Allow subquery results which are exec params as GapFill arguments * [#9825](#9825) Support `ADD COLUMN` on continuous aggregates * [#9842](#9842) Suppress continuous aggregate invalidation tracking during bulk loads * [#9878](#9878) Remove `chunk_constraint` catalog tracking for foreign keys * [#9893](#9893) Remove `chunk_constraint` catalog tracking for non-dimensional constraints * [#9903](#9903) Incremental refresh for `refresh_continuous_aggregate()` * [#9915](#9915) Remove `_timescaledb_catalog.chunk_constraint` table * [#9938](#9938) Add `rebuild_sparse_index` function * [#9964](#9964) Add a function to lock OSM chunk's dimension slice * [#9980](#9980) Support `first/last(value, time)` in `ColumnarIndexScan` **Bugfixes** * [#9708](#9708) Guard time bucket parameter handling against bad input * [#9745](#9745) Check constraints when adding unique constraints to chunks * [#9890](#9890) Fix incremental refresh batch boundaries to align with variable-width buckets and start only where a chunk and an invalidation overlap * [#9914](#9914) Fix use-after-free in segmentwise recompression * [#9929](#9929) Fix background jobs being bumped in the queue forever and never running * [#9955](#9955) Fix wrong results when using Batch Sorted Merge with no first-last index on a non-leading order by column * [#9967](#9967) Block upgrade after downgrade with first/last indexes present * [#9976](#9976) Fix wrong results when comparing a date column to a `timestamptz` value * [#9977](#9977) Fix `COPY WHERE` into a hypertable with dropped columns * [#9981](#9981) Fix set-returning functions in the sort key of `ColumnarScan` * [#9982](#9982) Reject `ALTER TABLE ... INHERIT` when the parent is a hypertable * [#9984](#9984) Fix handling of `NOT VALID NOT NULL` constraint for query optimization * [#9986](#9986) Handle `MERGE WHEN NOT MATCHED BY SOURCE` on hypertables * [#9988](#9988) Fix `time_bucket_gapfill` function detection * [#10003](#10003) Block unsafe updates of unique columns on compressed chunks * [#10024](#10024) Fix `approximate_row_count` handling of Infinity * [#10025](#10025) Fix rename on compressed continuous aggregates * [#10026](#10026) Fix chunk skipping near `PG_INT64_MAX` **New Settings** * `skip_cagg_invalidation`: skip continuous aggregate invalidation tracking for DML and DDL in the current session/transaction. Off by default. * `stats_max_chunks`: set the per-database compressed chunk statistics cache capacity. Defaults to 1024 chunks; set to 0 to disable the feature. **Thanks** * @Fabian-2596 for suggesting more accurate GapFill row count estimate * @otjdiepluong for fixing spelling mistakes in timescaledb source code comments * @scimad and @Nosfistis for suggesting expanding coverage for gapfill arguments

github-actions Bot assigned melihmutlu May 26, 2026

melihmutlu force-pushed the incremental-manual-refresh branch from 8af7edb to f051a1f Compare May 26, 2026 14:06

melihmutlu changed the title ~~Incremental manual refresh~~ Incremental refresh for refresh_continuous_aggregate() May 26, 2026

melihmutlu force-pushed the incremental-manual-refresh branch from 1ee0f89 to 7ba4d11 Compare May 26, 2026 14:53

melihmutlu marked this pull request as ready for review May 26, 2026 14:53

melihmutlu requested a review from a team May 26, 2026 14:54

melihmutlu added the Continuous Aggregate label May 26, 2026

github-actions Bot requested review from Poroma-Banerjee and pnthao May 26, 2026 14:55

pnthao reviewed May 26, 2026

View reviewed changes

Comment thread tsl/src/continuous_aggs/refresh.c

gayyappan requested a review from a team May 27, 2026 12:58

vineethapai added this to the v2.28.0 milestone Jun 1, 2026

melihmutlu force-pushed the incremental-manual-refresh branch 4 times, most recently from 8c559aa to 2f7e495 Compare June 5, 2026 15:24

pnthao reviewed Jun 5, 2026

View reviewed changes

melihmutlu force-pushed the incremental-manual-refresh branch from 2f7e495 to 2db2db7 Compare June 5, 2026 20:12

kpan2034 reviewed Jun 8, 2026

View reviewed changes

Comment thread tsl/test/isolation/specs/cagg_refresh_cleanup_register.spec Outdated

kpan2034 approved these changes Jun 8, 2026

View reviewed changes

melihmutlu force-pushed the incremental-manual-refresh branch 5 times, most recently from 35b4482 to e3f5c9d Compare June 8, 2026 12:32

pnthao self-requested a review June 8, 2026 17:56

pnthao approved these changes Jun 8, 2026

View reviewed changes

gayyappan reviewed Jun 8, 2026

View reviewed changes

Comment thread tsl/src/continuous_aggs/refresh.c

melihmutlu force-pushed the incremental-manual-refresh branch 2 times, most recently from c4f39b3 to e5b97b6 Compare June 8, 2026 19:26

melihmutlu enabled auto-merge (squash) June 8, 2026 19:40

melihmutlu force-pushed the incremental-manual-refresh branch from e5b97b6 to 5586fdf Compare June 8, 2026 19:49

melihmutlu merged commit 40421b9 into timescale:main Jun 8, 2026
56 of 57 checks passed

surister mentioned this pull request Jun 9, 2026

Release 2.28.0 #10010

Merged

melihmutlu deleted the incremental-manual-refresh branch June 9, 2026 14:50

atovpeko mentioned this pull request Jun 11, 2026

Document incremental refresh_continuous_aggregate() options (2.28.0) timescale/Tiger-Data-Docs#315

Merged

7 tasks

timescale-automation added the released-2.28.0 Released in 2.28.0 label Jun 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Incremental refresh for refresh_continuous_aggregate() - #9903

Incremental refresh for refresh_continuous_aggregate()#9903
melihmutlu merged 1 commit into
timescale:mainfrom
melihmutlu:incremental-manual-refresh

melihmutlu commented May 26, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 26, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpan2034 left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

melihmutlu commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented May 26, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kpan2034 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

melihmutlu commented May 26, 2026 •

edited

Loading

codecov Bot commented May 26, 2026 •

edited

Loading