Improve gapfill row count estimate by natalya-aksman · Pull Request #9735 · timescale/timescaledb

natalya-aksman · 2026-05-07T19:08:21Z

Fixes #9716: bad Gapfill row count estimate of 0 (due to 0 original rows which ignores all the gap fills generating a lot of rows) leading to bad plans and bad performance.
It's not a bug fix but an enhancement.

We can estimate Gapfill row count much better if we can evaluate start/finish/period at planning time and use more accurate row count estimate of (subplan rows) + (finish-start)/period instead of just subplan rows.

We may have maximum of original rows plus filled gaps in addition to those rows, and this is the estimate used here.

github-actions · 2026-05-07T19:08:45Z

@melihmutlu, @antekresic: please review this pull request.

Powered by pull-review

codecov · 2026-05-07T19:45:25Z

Codecov Report

❌ Patch coverage is 97.65625% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tsl/src/nodes/gapfill/gapfill_plan.c	97.29%	0 Missing and 2 partials ⚠️
tsl/src/nodes/gapfill/gapfill_exec.c	98.14%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

akuzm · 2026-05-08T13:29:44Z

 DEALLOCATE coalesce_tz_test;
 RESET plan_cache_mode;
 DROP TABLE gf_tz_test;
+-- Gapfill row count estimate should be ~((finish - start)/period + 1) instead of subpath row count,


Good idea, just thought about this today in relation to plain time_bucket grouping cardinality. Maybe we'd be able to apply this approach there too.

natalya-aksman · 2026-05-13T15:52:05Z

+ORDER BY 1;
+:PREFIX EXECUTE prep_int3(1,0,100);
+--- QUERY PLAN ---
+ Custom Scan (GapFill)  (cost=1.10..1.13 rows=1 width=0) (actual rows=100.00 loops=1)


When we have generic plan and can't evaluate gapfill boundaries at planning time we fall back to subplan->rows, see above. We estimate rows = 1 from the subplan while actual rows are 100.

svenklemm · 2026-05-19T10:56:39Z

-	path->cpath.path.rows = subpath->rows;
+	/* we may have filled gaps in addition to original tuples
+	 * filled gaps set per each group */
+	path->cpath.path.rows = subpath->rows + gapfill_rows * num_groups;


shouldn't the formula be gapfill_rows * num_groups because subpath->rows will be merged in the gapfill groups due to the aggregates.

lets say you have january 1 to january 31 with 1 group so gapfill_rows should be 31. no matter how many rows are coming from subpath->rows you will always produce 31 rows

We can have gapfill window outside of existing data, which is the case in the bug report. In that case we will have suplan rows plus rows in the gapfill window. We are producing upper estimate i.e. we estimate max possible rows which will be in this scenario i.e. when we have all subplan rows outside of gapfill window plus gapfills in the gapfill window.

See unit test described by this comment.

natalya-aksman · 2026-05-21T17:07:27Z

+GROUP BY 1
+ORDER BY 1;
+--- QUERY PLAN ---
+ Custom Scan (GapFill)  (cost=1.20..1.21 rows=24 width=0) (actual rows=24.00 loops=1)


An example when we have 4 subplan rows between (-100,100) outside of gapfill window plus 20 gapfill rows in [200,300) gapfill window, the estimate of 24 rows is the same as actual rows.

* Rebuild Postgres caches weekly instead of daily (timescale#9892) The daily rebuilds are apparently creating too much of the cache load, we're going over the GitHub limit of 10GB, and useful caches like the libfuzzer examples start to get evicted. I think weekly rebuilds also give a reasonable time frame to detect the Postgres builds breaking due to environmen changes. * Improve gapfill row count estimate (timescale#9735) Fixes timescale#9716: bad Gapfill row count estimate of 0 (due to 0 original rows which ignores all the gap fills generating a lot of rows) leading to bad plans and bad performance. It's not a bug fix but an enhancement. We can estimate Gapfill row count much better if we can evaluate start/finish/period at planning time and use more accurate row count estimate of `(subplan rows) + (finish-start)/period` instead of just `subplan rows`. We may have maximum of original rows plus filled gaps in addition to those rows, and this is the estimate used here. * Bump PG used for Windows CI to 18.4, 17.10, 16.14 and 15.18 Latest PG packages for windows weren't available when we switched the rest of CI to latest version. * Update dependencies * Make ordering in update test deterministic psql \d does not order constraints with same name deterministically so replace \d with equivalent catalog queries * Drop chunk-side foreign keys via event trigger The previous approach linked each chunk-side foreign key to its hypertable parent with a DEPENDENCY_AUTO edge in pg_depend so DROP and RENAME on the hypertable would cascade. pg_dump does not dump pg_depend rows, so the edge was gone after a restore and the chunk constraints survived. * Update codecov/codecov-action action to v6.0.1 * prepare 2.28.0 release ommitting not in ci checks stuff on purpose

@Fabian-2596

## 2.28.0 (2026-06-16) This release contains performance improvements and bug fixes since the 2.27.2 release. We recommend that you upgrade at the next available opportunity. **Highlighted features in TimescaleDB v2.28.0** * **Faster `first()` and `last()` queries on compressed data.** TimescaleDB derives `first(value, time)` and `last(value, time)` aggregates straight from the columnstore's batch metadata, skipping batch decompression entirely. For the "latest reading per series" lookups that time-series workloads run constantly, that means meaningfully faster recency queries with no changes to your SQL queries. * **Lighter, less disruptive continuous aggregate refreshes.** `refresh_continuous_aggregate()` can now run incrementally in batches — the same behavior refresh policies already use — enabling breaking large manual refreshes into smaller chunks (tunable via `buckets_per_batch`, `max_batches_per_execution`, and `refresh_newest_first`) instead of one heavy operation. Refreshes also now take a lighter lock while processing the invalidation log, so they no longer block unrelated concurrent operations on the same continuous aggregate, improving behavior for concurrent workloads. * **Vectorized execution now covers `CASE` expressions.** TimescaleDB's columnar executor can now evaluate `CASE ... WHEN` expressions directly on compressed data, so queries using conditional logic stay on the fast vectorized path instead of falling back to slower row-by-row decompression. This speeds up a common pattern — conditional aggregations and computed columns over compressed history — with no query changes needed. * **Add new aggregations to a continuous aggregate without rebuilding it.** You can now run `ALTER MATERIALIZED VIEW <cagg> ADD COLUMN <name> <type> GENERATED ALWAYS AS (<aggregate>) STORED` to add a new computed aggregate to an existing continuous aggregate in place — no more dropping and recreating the whole aggregate just to track one more metric. New data populates the column going forward, letting your rollups evolve alongside your application. (Existing rows start as `NULL`; a forced refresh backfills them when you need historical values.) **Deprecation Notice: PostgreSQL 15 Support** This release marks the final minor version of TimescaleDB that will support PostgreSQL 15. Starting with our next release, version 2.29.0, we will officially drop support for Postgres 15, and only support Postgres 16, 17, and 18; however, all future patch releases within the current 2.28 version cycle will continue to fully support it. We recommend planning your PostgreSQL upgrades accordingly to ensure a smooth transition. **Deprecation Notice: `chunk_constraint` Catalog Table** Please note that the `_timescaledb_catalog.chunk_constraint` table has been dropped and temporarily replaced by a view, which introduces a change to the underlying objects while maintaining current query behavior. However, this compatibility view will be completely removed in a future release. To ensure your queries remain compatible moving forward, we strongly advise transitioning to the stable contracts provided by our [informational views](https://www.tigerdata.com/docs/reference/timescaledb/informational-views). **Backward-Incompatible Changes** * [#9934](#9934) Remove adaptive chunking **Features** * [#4054](#4054) Support `ANALYZE` and `VACUUM` on continuous aggregates by redirecting to the underlying materialization hypertable * [#9125](#9125) Increase the parallelism of `SELECT` queries over compressed hypertables to approximately match the uncompressed data size * [#9410](#9410) Mark `hypertable` and `chunk` as user catalog tables * [#9416](#9416) Support some forms of `CASE` expression in columnar aggregation and grouping * [#9580](#9580) Add `first` / `last` sparse indexes to compression * [#9784](#9784) Use `first` / `last` sparse index for `orderby` metadata on new compressed chunks * [#9668](#9668) Allow database owner to configure hypertables and policies * [#9701](#9701) Relax lock during continuous aggregate invalidation log processing * [#9730](#9730) Add in-memory observability for compressed chunks * [#9735](#9735) Improve `GapFill` row count estimate * [#9821](#9821) Allow subquery results which are exec params as GapFill arguments * [#9825](#9825) Support `ADD COLUMN` on continuous aggregates * [#9842](#9842) Suppress continuous aggregate invalidation tracking during bulk loads * [#9878](#9878) Remove `chunk_constraint` catalog tracking for foreign keys * [#9893](#9893) Remove `chunk_constraint` catalog tracking for non-dimensional constraints * [#9903](#9903) Incremental refresh for `refresh_continuous_aggregate()` * [#9915](#9915) Remove `_timescaledb_catalog.chunk_constraint` table * [#9938](#9938) Add `rebuild_sparse_index` function * [#9964](#9964) Add a function to lock OSM chunk's dimension slice * [#9980](#9980) Support `first/last(value, time)` in `ColumnarIndexScan` **Bugfixes** * [#9708](#9708) Guard time bucket parameter handling against bad input * [#9745](#9745) Check constraints when adding unique constraints to chunks * [#9890](#9890) Fix incremental refresh batch boundaries to align with variable-width buckets and start only where a chunk and an invalidation overlap * [#9914](#9914) Fix use-after-free in segmentwise recompression * [#9929](#9929) Fix background jobs being bumped in the queue forever and never running * [#9955](#9955) Fix wrong results when using Batch Sorted Merge with no first-last index on a non-leading order by column * [#9967](#9967) Block upgrade after downgrade with first/last indexes present * [#9976](#9976) Fix wrong results when comparing a date column to a `timestamptz` value * [#9977](#9977) Fix `COPY WHERE` into a hypertable with dropped columns * [#9981](#9981) Fix set-returning functions in the sort key of `ColumnarScan` * [#9982](#9982) Reject `ALTER TABLE ... INHERIT` when the parent is a hypertable * [#9984](#9984) Fix handling of `NOT VALID NOT NULL` constraint for query optimization * [#9986](#9986) Handle `MERGE WHEN NOT MATCHED BY SOURCE` on hypertables * [#9988](#9988) Fix `time_bucket_gapfill` function detection * [#10003](#10003) Block unsafe updates of unique columns on compressed chunks * [#10024](#10024) Fix `approximate_row_count` handling of Infinity * [#10025](#10025) Fix rename on compressed continuous aggregates * [#10026](#10026) Fix chunk skipping near `PG_INT64_MAX` **New Settings** * `skip_cagg_invalidation`: skip continuous aggregate invalidation tracking for DML and DDL in the current session/transaction. Off by default. * `stats_max_chunks`: set the per-database compressed chunk statistics cache capacity. Defaults to 1024 chunks; set to 0 to disable the feature. **Thanks** * @Fabian-2596 for suggesting more accurate GapFill row count estimate * @otjdiepluong for fixing spelling mistakes in timescaledb source code comments * @scimad and @Nosfistis for suggesting expanding coverage for gapfill arguments

natalya-aksman requested a review from a team May 7, 2026 19:08

github-actions Bot assigned natalya-aksman May 7, 2026

github-actions Bot requested review from antekresic and melihmutlu May 7, 2026 19:08

natalya-aksman added gapfill enhancement An enhancement to an existing feature for functionality labels May 7, 2026

natalya-aksman added this to the v2.28.0 milestone May 7, 2026

natalya-aksman force-pushed the improve_gapfill_rowcount_estimate branch from d7e5fec to 880bc38 Compare May 7, 2026 19:34

natalya-aksman requested review from Poroma-Banerjee and akuzm and removed request for melihmutlu May 7, 2026 19:56

akuzm reviewed May 8, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_exec.c Outdated

akuzm reviewed May 8, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 8, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 8, 2026

View reviewed changes

natalya-aksman force-pushed the improve_gapfill_rowcount_estimate branch from 880bc38 to 22a0566 Compare May 8, 2026 14:32

natalya-aksman requested a review from svenklemm May 11, 2026 13:51

natalya-aksman commented May 13, 2026

View reviewed changes

natalya-aksman force-pushed the improve_gapfill_rowcount_estimate branch 2 times, most recently from 3ea80f2 to b9e82bd Compare May 13, 2026 17:09

akuzm reviewed May 13, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 13, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 13, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 13, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_exec.c Outdated

natalya-aksman force-pushed the improve_gapfill_rowcount_estimate branch 3 times, most recently from 3ba33fa to 5df1f04 Compare May 15, 2026 20:49

svenklemm reviewed May 19, 2026

View reviewed changes

natalya-aksman commented May 21, 2026

View reviewed changes

akuzm reviewed May 22, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c Outdated

akuzm reviewed May 22, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_exec.c Outdated

akuzm reviewed May 22, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_exec.c Outdated

akuzm reviewed May 22, 2026

View reviewed changes

Comment thread tsl/src/nodes/gapfill/gapfill_plan.c

akuzm approved these changes May 22, 2026

View reviewed changes

Improve gapfill row count estimate

eadbf58

natalya-aksman force-pushed the improve_gapfill_rowcount_estimate branch from 5df1f04 to eadbf58 Compare May 22, 2026 17:38

Poroma-Banerjee approved these changes May 22, 2026

View reviewed changes

natalya-aksman merged commit 44ef2da into timescale:main May 22, 2026
56 of 60 checks passed

surister mentioned this pull request Jun 9, 2026

Release 2.28.0 #10010

Merged

timescale-automation added the released-2.28.0 Released in 2.28.0 label Jun 16, 2026

Uh oh!

Conversation

natalya-aksman commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 7, 2026

Uh oh!

codecov Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

akuzm May 8, 2026

Choose a reason for hiding this comment

Uh oh!

natalya-aksman May 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

svenklemm May 19, 2026

Choose a reason for hiding this comment

Uh oh!

svenklemm May 19, 2026

Choose a reason for hiding this comment

Uh oh!

natalya-aksman May 19, 2026

Choose a reason for hiding this comment

Uh oh!

natalya-aksman May 21, 2026

Choose a reason for hiding this comment

Uh oh!

natalya-aksman May 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

natalya-aksman commented May 7, 2026 •

edited

Loading

codecov Bot commented May 7, 2026 •

edited

Loading