Stats collection for compressed chunks by dbeck · Pull Request #9730 · timescale/timescaledb

dbeck · 2026-05-06T17:53:22Z

This change collects information about compressed chunks and stores
them in an LRU fashion. It captures events at compression, query
and DML oparations. The query/DML operations maintain totals and
information about the last operation.

The chunk information is maintained in shared memory per database.
One can access the data via calling the

_timescaledb_functions.chunk_statistics(..) function or by

selecting from the

timescaledb_information.stat_chunk_activity view

the former allows the user to push down filters, so it is faster and with
less lock contention.

The amount of shared memory is controlled by the GUC:

timescaledb.stats_max_chunks

If the GUC is set to zero, the feature is disabled. The default
value is 1024 and the maximum is 262144. The amount must be a
power of two number. The stats_max_chunks value controls
how many chunk we store information about. The memory consumption
is 300 bytes per entry.

If the loader is not upgraded, the statistics will be disabled on PG 15/16.
PG 17 doesn't require a loader change, so we can upgrade the extension
without restarting:

Disable-check: loader-change

Co-authored-by: Ante Kresic antekresic@users.noreply.github.com
Signed-off-by: David Beck dbeck@users.noreply.github.com

github-actions · 2026-05-06T17:53:45Z

@kpan2034, @pnthao: please review this pull request.

Powered by pull-review

codecov · 2026-05-06T18:59:47Z

Codecov Report

❌ Patch coverage is 74.62687% with 170 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/ts_stats/ts_stats_srf.c	64.39%	62 Missing and 11 partials ⚠️
src/ts_stats/ts_stats_record.c	73.88%	26 Missing and 15 partials ⚠️
src/ts_stats/ts_stats_segment.c	77.21%	20 Missing and 16 partials ⚠️
src/guc.c	25.00%	8 Missing and 4 partials ⚠️
src/ts_stats/ts_stats_defs.h	90.32%	0 Missing and 6 partials ⚠️
tsl/src/compression/compression.c	95.23%	0 Missing and 1 partial ⚠️
tsl/src/nodes/columnar_scan/compressed_batch.c	83.33%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

fabriziomello · 2026-05-06T19:51:24Z

Hey @dbeck, not sure if you're aware (and if fit with this PR) but Postgres has Custom Cumulative Statistics for extensions since PG18 - https://wiki.postgresql.org/wiki/CustomCumulativeStats. Those custom cumulatiuve statistcs survive to restarts like any other Postgres cumulative statistic (aka pg_stat_*).

dbeck · 2026-05-07T08:51:19Z

Hey @dbeck, not sure if you're aware (and if fit with this PR) but Postgres has Custom Cumulative Statistics for extensions since PG18 - https://wiki.postgresql.org/wiki/CustomCumulativeStats. Those custom cumulatiuve statistcs survive to restarts like any other Postgres cumulative statistic (aka pg_stat_*).

Hey @fabriziomello , good to hear from you! Thanks for the pointer, I quickly checked and I'll look more into it. I see it has lots of bells and whistles, looks quite a bit complicated to achieve what we wanted. The support of previous versions is a concern too. I'll discuss with the team if this acceptable to only support PG18.

antekresic

Another pass, will look at this some more.

antekresic

Couple of nits but otherwise it lgtm!

This change collects information about compressed chunks and stores them in an LRU fashion. It captures events at compression, query and DML oparations. The query/DML operations maintain totals and information about the last operation. The chunk information is maintained in shared memory per database. One can access the data via calling the - `_timescaledb_functions.chunk_statistics(..)` function or by selecting from the - `timescaledb_information.stat_chunk_activity` view the former allows the user to push down filters, so it is faster and with less lock contention. The amount of shared memory is controlled by the GUC: - `timescaledb.stats_max_chunks` If the GUC is set to zero, the feature is disabled. The default value is 1024 and the maximum is 262144. The amount must be a power of two number. The `stats_max_chunks` value controls how many chunk we store information about. The memory consumption is 300 bytes per entry. If the loader is not upgraded, the statistics will be disabled on PG 15/16. PG 17 doesn't require a loader change, so we can upgrade the extension without restarting: Disable-check: loader-change Co-authored-by: Ante Kresic <antekresic@users.noreply.github.com> Signed-off-by: David Beck <dbeck@users.noreply.github.com>

@Fabian-2596

## 2.28.0 (2026-06-16) This release contains performance improvements and bug fixes since the 2.27.2 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.28.0** * **Faster `first()` and `last()` queries on compressed data.** TimescaleDB derives `first(value, time)` and `last(value, time)` aggregates straight from the columnstore's batch metadata, skipping batch decompression entirely. For the "latest reading per series" lookups that time-series workloads run constantly, that means meaningfully faster recency queries with no changes to your SQL queries. * **Lighter, less disruptive continuous aggregate refreshes.** `refresh_continuous_aggregate()` can now run incrementally in batches — the same behavior refresh policies already use — enabling breaking large manual refreshes into smaller chunks (tunable via `buckets_per_batch`, `max_batches_per_execution`, and `refresh_newest_first`) instead of one heavy operation. Refreshes also now take a lighter lock while processing the invalidation log, so they no longer block unrelated concurrent operations on the same continuous aggregate, improving behavior for concurrent workloads. * **Vectorized execution now covers `CASE` expressions.** TimescaleDB's columnar executor can now evaluate `CASE ... WHEN` expressions directly on compressed data, so queries using conditional logic stay on the fast vectorized path instead of falling back to slower row-by-row decompression. This speeds up a common pattern — conditional aggregations and computed columns over compressed history — with no query changes needed. * **Add new aggregations to a continuous aggregate without rebuilding it.** You can now run `ALTER MATERIALIZED VIEW <cagg> ADD COLUMN <name> <type> GENERATED ALWAYS AS (<aggregate>) STORED` to add a new computed aggregate to an existing continuous aggregate in place — no more dropping and recreating the whole aggregate just to track one more metric. New data populates the column going forward, letting your rollups evolve alongside your application. (Existing rows start as `NULL`; a forced refresh backfills them when you need historical values.) **Deprecation Notice: PostgreSQL 15 Support** This release marks the final minor version of TimescaleDB that will support PostgreSQL 15. Starting with our next release, version 2.29.0, we will officially drop support for Postgres 15, and only support Postgres 16, 17, and 18; however, all future patch releases within the current 2.28 version cycle will continue to fully support it. We recommend planning your PostgreSQL upgrades accordingly to ensure a smooth transition. **Deprecation Notice: `chunk_constraint` Catalog Table** Please note that the `_timescaledb_catalog.chunk_constraint` table has been dropped and temporarily replaced by a view, which introduces a change to the underlying objects while maintaining current query behavior. However, this compatibility view will be completely removed in a future release. To ensure your queries remain compatible moving forward, we strongly advise transitioning to the stable contracts provided by our [informational views](https://www.tigerdata.com/docs/reference/timescaledb/informational-views). **Backward-Incompatible Changes** * [#9934](#9934) Remove adaptive chunking **Features** * [#4054](#4054) Support `ANALYZE` and `VACUUM` on continuous aggregates by redirecting to the underlying materialization hypertable * [#9125](#9125) Increase the parallelism of `SELECT` queries over compressed hypertables to approximately match the uncompressed data size * [#9410](#9410) Mark `hypertable` and `chunk` as user catalog tables * [#9416](#9416) Support some forms of `CASE` expression in columnar aggregation and grouping * [#9580](#9580) Add `first` / `last` sparse indexes to compression * [#9784](#9784) Use `first` / `last` sparse index for `orderby` metadata on new compressed chunks * [#9668](#9668) Allow database owner to configure hypertables and policies * [#9701](#9701) Relax lock during continuous aggregate invalidation log processing * [#9730](#9730) Add in-memory observability for compressed chunks * [#9735](#9735) Improve `GapFill` row count estimate * [#9821](#9821) Allow subquery results which are exec params as GapFill arguments * [#9825](#9825) Support `ADD COLUMN` on continuous aggregates * [#9842](#9842) Suppress continuous aggregate invalidation tracking during bulk loads * [#9878](#9878) Remove `chunk_constraint` catalog tracking for foreign keys * [#9893](#9893) Remove `chunk_constraint` catalog tracking for non-dimensional constraints * [#9903](#9903) Incremental refresh for `refresh_continuous_aggregate()` * [#9915](#9915) Remove `_timescaledb_catalog.chunk_constraint` table * [#9938](#9938) Add `rebuild_sparse_index` function * [#9964](#9964) Add a function to lock OSM chunk's dimension slice * [#9980](#9980) Support `first/last(value, time)` in `ColumnarIndexScan` **Bugfixes** * [#9708](#9708) Guard time bucket parameter handling against bad input * [#9745](#9745) Check constraints when adding unique constraints to chunks * [#9890](#9890) Fix incremental refresh batch boundaries to align with variable-width buckets and start only where a chunk and an invalidation overlap * [#9914](#9914) Fix use-after-free in segmentwise recompression * [#9929](#9929) Fix background jobs being bumped in the queue forever and never running * [#9955](#9955) Fix wrong results when using Batch Sorted Merge with no first-last index on a non-leading order by column * [#9967](#9967) Block upgrade after downgrade with first/last indexes present * [#9976](#9976) Fix wrong results when comparing a date column to a `timestamptz` value * [#9977](#9977) Fix `COPY WHERE` into a hypertable with dropped columns * [#9981](#9981) Fix set-returning functions in the sort key of `ColumnarScan` * [#9982](#9982) Reject `ALTER TABLE ... INHERIT` when the parent is a hypertable * [#9984](#9984) Fix handling of `NOT VALID NOT NULL` constraint for query optimization * [#9986](#9986) Handle `MERGE WHEN NOT MATCHED BY SOURCE` on hypertables * [#9988](#9988) Fix `time_bucket_gapfill` function detection * [#10003](#10003) Block unsafe updates of unique columns on compressed chunks * [#10024](#10024) Fix `approximate_row_count` handling of Infinity * [#10025](#10025) Fix rename on compressed continuous aggregates * [#10026](#10026) Fix chunk skipping near `PG_INT64_MAX` **New Settings** * `skip_cagg_invalidation`: skip continuous aggregate invalidation tracking for DML and DDL in the current session/transaction. Off by default. * `stats_max_chunks`: set the per-database compressed chunk statistics cache capacity. Defaults to 1024 chunks; set to 0 to disable the feature. **Thanks** * @Fabian-2596 for suggesting more accurate GapFill row count estimate * @otjdiepluong for fixing spelling mistakes in timescaledb source code comments * @scimad and @Nosfistis for suggesting expanding coverage for gapfill arguments

dbeck requested review from a team and antekresic May 6, 2026 17:53

github-actions Bot assigned dbeck May 6, 2026

github-actions Bot requested review from kpan2034 and pnthao May 6, 2026 17:53

dbeck force-pushed the dbeck/observ_next branch 5 times, most recently from 5ef3b89 to 355039c Compare May 6, 2026 18:48

dbeck force-pushed the dbeck/observ_next branch from 355039c to 4c71109 Compare May 7, 2026 08:17

dbeck force-pushed the dbeck/observ_next branch 2 times, most recently from dd63020 to 7c24bb7 Compare May 7, 2026 20:27

dbeck changed the title ~~Observability for chunk data~~ Stats collection for compressed chunks May 8, 2026

dbeck force-pushed the dbeck/observ_next branch 4 times, most recently from 4c525fd to 2c138c5 Compare May 8, 2026 12:48

antekresic reviewed May 13, 2026

View reviewed changes

Comment thread src/guc.c Outdated

Comment thread test/expected/observ_funcs.out Outdated

Comment thread src/guc.h Outdated

Comment thread src/ts_stats/ts_stats_srf.c Outdated

dbeck force-pushed the dbeck/observ_next branch from 2c138c5 to 7eaa081 Compare May 13, 2026 13:16

dbeck force-pushed the dbeck/observ_next branch 6 times, most recently from 14b87a1 to f6d250f Compare May 20, 2026 18:45

antekresic self-requested a review May 21, 2026 10:29

antekresic reviewed May 21, 2026

View reviewed changes

Comment thread src/ts_stats/ts_stats_srf.c

Comment thread src/guc.c Outdated

Comment thread src/ts_stats/ts_stats_srf.c Outdated

Comment thread src/ts_stats/ts_stats_segment.h Outdated

Comment thread src/ts_stats/ts_stats_defs.h Outdated

Comment thread src/ts_stats/ts_stats_segment.c

dbeck force-pushed the dbeck/observ_next branch from 784fbe7 to 42dee6c Compare May 21, 2026 15:01

antekresic approved these changes May 26, 2026

View reviewed changes

Comment thread tsl/src/compression/compression.h

Comment thread src/ts_stats/ts_stats_defs.h Outdated

dbeck force-pushed the dbeck/observ_next branch 3 times, most recently from ed9be87 to 3473922 Compare May 26, 2026 14:35

natalya-aksman reviewed May 26, 2026

View reviewed changes

Comment thread tsl/test/expected/compress_observ-18.out

dbeck force-pushed the dbeck/observ_next branch 5 times, most recently from 2a8c441 to 766b6d1 Compare May 26, 2026 18:29

Poroma-Banerjee approved these changes May 27, 2026

View reviewed changes

Comment thread tsl/src/compression/compression.c

dbeck force-pushed the dbeck/observ_next branch from 766b6d1 to 52cdb82 Compare May 27, 2026 10:19

dbeck force-pushed the dbeck/observ_next branch from 52cdb82 to bf9b5e6 Compare May 27, 2026 10:27

natalya-aksman approved these changes May 27, 2026

View reviewed changes

dbeck merged commit d801f80 into main May 27, 2026
62 of 64 checks passed

dbeck deleted the dbeck/observ_next branch May 27, 2026 14:51

dbeck added this to the v2.28.0 milestone Jun 3, 2026

surister mentioned this pull request Jun 9, 2026

Release 2.28.0 #10010

Merged

timescale-automation added the released-2.28.0 Released in 2.28.0 label Jun 16, 2026

Uh oh!

Conversation

dbeck commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 6, 2026

Uh oh!

codecov Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fabriziomello commented May 6, 2026

Uh oh!

dbeck commented May 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antekresic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

antekresic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dbeck commented May 6, 2026 •

edited

Loading

codecov Bot commented May 6, 2026 •

edited

Loading