Skip to content

Commit 5ebc0fa

Browse files
melihmutlumelihmutlu
authored
Prune real-time branch of deep hierarchical caggs (#10175)
A single application of ts_transform_time_bucket_comparison() strips only the outermost time_bucket() from a comparison. In a hierarchical continuous aggregate the bucket input is itself another time_bucket() call. The real-time branch could not be proven empty and the refresh re-scanned the whole raw un-materialized tail of the base hypertable. Add ts_transform_nested_time_bucket_comparison(), which applies the single-level transform repeatedly until the bound is expressed on the raw column. Only the original qual and the fully reduced qual are kept; the intermediate levels have no consumer. Fixes: #10071
1 parent d776703 commit 5ebc0fa

10 files changed

Lines changed: 638 additions & 4 deletions

‎.unreleased/pr_10175‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Fixes: #10071 Prune the real-time branch of hierarchical continuous aggregates at any nesting depth
2+
Thanks: @viniciusrsouza for reporting the issue

‎src/nodes/chunk_append/exec.c‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1001,7 +1001,7 @@ ts_constify_restrictinfos(PlannerInfo *root, List *restrictinfos)
10011001
* transformations again. This might allow us to exclude chunks
10021002
* based on a parameterized time_bucket expression.
10031003
*/
1004-
Expr *additional_clause = ts_transform_time_bucket_comparison(constified);
1004+
Expr *additional_clause = ts_transform_nested_time_bucket_comparison(constified);
10051005
if (additional_clause != NULL)
10061006
{
10071007
/*

‎src/planner/expand_hypertable.c‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -731,6 +731,32 @@ ts_transform_time_bucket_comparison(Expr *node)
731731
return &op->xpr;
732732
}
733733

734+
/*
735+
* Fully unwrap a (possibly nested) time_bucket() comparison by applying the
736+
* single-level transform repeatedly until the bound reaches the raw column.
737+
*
738+
* ts_transform_time_bucket_comparison() only strips the outermost time_bucket().
739+
* In hierarchical continuous aggregates the time_bucket() input column can be
740+
* itself another time_bucket() call.
741+
*
742+
* Used both during hypertable expansion and from the chunk-append executor's
743+
* runtime constification.
744+
*/
745+
Expr *
746+
ts_transform_nested_time_bucket_comparison(Expr *qual)
747+
{
748+
Expr *nested = ts_transform_time_bucket_comparison(qual);
749+
Expr *transformed = NULL;
750+
751+
while (nested != NULL)
752+
{
753+
transformed = nested;
754+
nested = ts_transform_time_bucket_comparison(transformed);
755+
}
756+
757+
return transformed;
758+
}
759+
734760
/*
735761
* Since baserestrictinfo is not yet set by the planner, we have to derive
736762
* it ourselves. It's safe for us to miss some restrict info clauses (this
@@ -784,7 +810,7 @@ process_quals(Node *quals, CollectQualCtx *ctx, bool is_outer_join)
784810
* check for time_bucket comparisons
785811
* time_bucket(Const, time_colum) > Const
786812
*/
787-
Expr *transformed = ts_transform_time_bucket_comparison(qual);
813+
Expr *transformed = ts_transform_nested_time_bucket_comparison(qual);
788814
if (transformed != NULL)
789815
{
790816
/*
@@ -835,7 +861,7 @@ timebucket_annotate(Node *quals, CollectQualCtx *ctx)
835861
* check for time_bucket comparisons
836862
* time_bucket(Const, time_colum) > Const
837863
*/
838-
Expr *transformed = ts_transform_time_bucket_comparison(qual);
864+
Expr *transformed = ts_transform_nested_time_bucket_comparison(qual);
839865
if (transformed != NULL)
840866
{
841867
/*
@@ -1054,7 +1080,7 @@ get_simplified_restrictions(PlannerInfo *root, List *restrictions)
10541080
* check for time_bucket comparisons
10551081
* time_bucket(Const, time_colum) > Const
10561082
*/
1057-
Expr *transformed = ts_transform_time_bucket_comparison(qual);
1083+
Expr *transformed = ts_transform_nested_time_bucket_comparison(qual);
10581084
if (transformed != NULL)
10591085
{
10601086
/*

‎src/planner/planner.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ extern void ts_plan_expand_hypertable_chunks(Hypertable *ht, PlannerInfo *root,
9797
bool include_osm);
9898
extern void ts_plan_expand_timebucket_annotate(PlannerInfo *root, RelOptInfo *rel);
9999
extern Expr *ts_transform_time_bucket_comparison(Expr *);
100+
extern Expr *ts_transform_nested_time_bucket_comparison(Expr *);
100101
extern Node *ts_constify_now(PlannerInfo *root, List *rtable, Node *node);
101102
extern void ts_planner_constraint_cleanup(PlannerInfo *root, RelOptInfo *rel);
102103
extern Node *ts_add_space_constraints(PlannerInfo *root, List *rtable, Node *node);

‎tsl/test/expected/cagg_hierarchical_realtime-16.out‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
-- This file and its contents are licensed under the Timescale License.
2+
-- Please see the included NOTICE for copyright information and
3+
-- LICENSE-TIMESCALE for a copy of the license.
4+
-- Regression test for #10071.
5+
--
6+
-- A real-time continuous aggregate (timescaledb.materialized_only = false) that
7+
-- is three or more levels deep in a hierarchy (a cagg on a cagg on a cagg) must
8+
-- prune its real-time branch when refreshed for a window entirely below the
9+
-- watermark, exactly as a two-level cagg does. A refresh reads the cagg's
10+
-- internal partial view, so EXPLAINing a below-watermark SELECT on that partial
11+
-- view reproduces the refresh scan. Before the fix the real-time branch was
12+
-- pruned only up to two levels; at three or more it re-scanned the whole
13+
-- un-materialized tail of the base hypertable and discarded every row.
14+
SET timezone TO PST8PDT;
15+
SET max_parallel_workers_per_gather TO 0;
16+
SET client_min_messages TO WARNING;
17+
CREATE TABLE conditions (ts timestamptz NOT NULL, value double precision NOT NULL);
18+
SELECT create_hypertable('conditions', by_range('ts', INTERVAL '7 days'));
19+
create_hypertable
20+
-------------------
21+
(1,t)
22+
23+
INSERT INTO conditions
24+
SELECT ts, 1.0
25+
FROM generate_series(TIMESTAMPTZ '2026-01-01', TIMESTAMPTZ '2026-04-29', INTERVAL '1 hour') AS ts;
26+
-- Level 1: 6-hour buckets on the raw hypertable
27+
CREATE MATERIALIZED VIEW cagg_l1
28+
WITH (timescaledb.continuous, timescaledb.materialized_only = false) AS
29+
SELECT time_bucket('6 hours', ts) AS ts, count(*) AS cnt, sum(value) AS sum_value
30+
FROM conditions GROUP BY 1 WITH NO DATA;
31+
-- Level 2: 1-day buckets on cagg_l1 (raw -> 6h -> 1day)
32+
CREATE MATERIALIZED VIEW cagg_l2
33+
WITH (timescaledb.continuous, timescaledb.materialized_only = false) AS
34+
SELECT time_bucket('1 day', ts) AS ts, sum(cnt) AS cnt, sum(sum_value) AS sum_value
35+
FROM cagg_l1 GROUP BY 1 WITH NO DATA;
36+
-- Level 3: 7-day buckets on cagg_l2 (raw -> 6h -> 1day -> 7day)
37+
CREATE MATERIALIZED VIEW cagg_l3
38+
WITH (timescaledb.continuous, timescaledb.materialized_only = false) AS
39+
SELECT time_bucket('7 days', ts) AS ts, sum(cnt) AS cnt, sum(sum_value) AS sum_value
40+
FROM cagg_l2 GROUP BY 1 WITH NO DATA;
41+
-- Level 4: 28-day buckets on cagg_l3 (raw -> 6h -> 1day -> 7day -> 28day)
42+
CREATE MATERIALIZED VIEW cagg_l4
43+
WITH (timescaledb.continuous, timescaledb.materialized_only = false) AS
44+
SELECT time_bucket('28 days', ts) AS ts, sum(cnt) AS cnt, sum(sum_value) AS sum_value
45+
FROM cagg_l3 GROUP BY 1 WITH NO DATA;
46+
-- Materialize everything up to 2026-04-01, leaving an un-materialized tail.
47+
CALL refresh_continuous_aggregate('cagg_l1', NULL, TIMESTAMPTZ '2026-04-01');
48+
CALL refresh_continuous_aggregate('cagg_l2', NULL, TIMESTAMPTZ '2026-04-01');
49+
CALL refresh_continuous_aggregate('cagg_l3', NULL, TIMESTAMPTZ '2026-04-01');
50+
CALL refresh_continuous_aggregate('cagg_l4', NULL, TIMESTAMPTZ '2026-04-01');
51+
-- A refresh reads the cagg's internal partial view, so EXPLAINing a
52+
-- below-watermark SELECT on that partial view reproduces the refresh scan. Look
53+
-- up each partial view name.
54+
SELECT format('%I.%I', partial_view_schema, partial_view_name) AS pv_l2
55+
FROM _timescaledb_catalog.continuous_agg WHERE user_view_name = 'cagg_l2' \gset
56+
SELECT format('%I.%I', partial_view_schema, partial_view_name) AS pv_l3
57+
FROM _timescaledb_catalog.continuous_agg WHERE user_view_name = 'cagg_l3' \gset
58+
SELECT format('%I.%I', partial_view_schema, partial_view_name) AS pv_l4
59+
FROM _timescaledb_catalog.continuous_agg WHERE user_view_name = 'cagg_l4' \gset
60+
\set PREFIX 'EXPLAIN (COSTS OFF, TIMING OFF, SUMMARY OFF)'
61+
-- For a window entirely below the watermark the real-time branch must be pruned
62+
-- at every depth: the plan should fold it to "One-Time Filter: false" and touch
63+
-- only the materialized hypertable, never the base-hypertable (_hyper_1_*)
64+
-- chunks.
65+
-- 2 levels (control: pruned before and after the fix)
66+
:PREFIX SELECT * FROM :pv_l2 WHERE ts >= TIMESTAMPTZ '2026-01-13' AND ts < TIMESTAMPTZ '2026-02-10';
67+
--- QUERY PLAN ---
68+
GroupAggregate
69+
Group Key: (time_bucket('@ 1 day'::interval, _hyper_2_19_chunk.ts))
70+
-> Sort
71+
Sort Key: (time_bucket('@ 1 day'::interval, _hyper_2_19_chunk.ts))
72+
-> Result
73+
-> Append
74+
-> Index Scan using _hyper_2_19_chunk__materialized_hypertable_2_ts_idx on _hyper_2_19_chunk
75+
Index Cond: ((ts < 'Tue Mar 31 23:00:00 2026 PDT'::timestamp with time zone) AND (ts >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (ts < 'Wed Feb 11 00:00:00 2026 PST'::timestamp with time zone))
76+
Filter: ((time_bucket('@ 1 day'::interval, ts) >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (time_bucket('@ 1 day'::interval, ts) < 'Tue Feb 10 00:00:00 2026 PST'::timestamp with time zone))
77+
-> HashAggregate
78+
Group Key: time_bucket('@ 6 hours'::interval, ts)
79+
-> Result
80+
One-Time Filter: false
81+
82+
-- 3 levels (the bug: real-time branch must be pruned)
83+
:PREFIX SELECT * FROM :pv_l3 WHERE ts >= TIMESTAMPTZ '2026-01-13' AND ts < TIMESTAMPTZ '2026-02-10';
84+
--- QUERY PLAN ---
85+
GroupAggregate
86+
Group Key: (time_bucket('@ 7 days'::interval, _hyper_3_21_chunk.ts))
87+
-> Sort
88+
Sort Key: (time_bucket('@ 7 days'::interval, _hyper_3_21_chunk.ts))
89+
-> Result
90+
-> Append
91+
-> Index Scan using _hyper_3_21_chunk__materialized_hypertable_3_ts_idx on _hyper_3_21_chunk
92+
Index Cond: ((ts < 'Tue Mar 31 17:00:00 2026 PDT'::timestamp with time zone) AND (ts >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (ts < 'Tue Feb 17 00:00:00 2026 PST'::timestamp with time zone))
93+
Filter: ((time_bucket('@ 7 days'::interval, ts) >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (time_bucket('@ 7 days'::interval, ts) < 'Tue Feb 10 00:00:00 2026 PST'::timestamp with time zone))
94+
-> HashAggregate
95+
Group Key: time_bucket('@ 1 day'::interval, (time_bucket('@ 6 hours'::interval, ts)))
96+
-> Result
97+
-> HashAggregate
98+
Group Key: time_bucket('@ 6 hours'::interval, ts)
99+
-> Result
100+
One-Time Filter: false
101+
102+
-- 4 levels
103+
:PREFIX SELECT * FROM :pv_l4 WHERE ts >= TIMESTAMPTZ '2026-01-13' AND ts < TIMESTAMPTZ '2026-02-10';
104+
--- QUERY PLAN ---
105+
GroupAggregate
106+
Group Key: (time_bucket('@ 28 days'::interval, _materialized_hypertable_4.ts))
107+
-> Sort
108+
Sort Key: (time_bucket('@ 28 days'::interval, _materialized_hypertable_4.ts))
109+
-> Result
110+
-> Append
111+
-> Append
112+
-> Index Scan using _hyper_4_22_chunk__materialized_hypertable_4_ts_idx on _hyper_4_22_chunk
113+
Index Cond: ((ts < 'Sun Mar 29 17:00:00 2026 PDT'::timestamp with time zone) AND (ts >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (ts < 'Tue Mar 10 01:00:00 2026 PDT'::timestamp with time zone))
114+
Filter: ((time_bucket('@ 28 days'::interval, ts) >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (time_bucket('@ 28 days'::interval, ts) < 'Tue Feb 10 00:00:00 2026 PST'::timestamp with time zone))
115+
-> Index Scan using _hyper_4_23_chunk__materialized_hypertable_4_ts_idx on _hyper_4_23_chunk
116+
Index Cond: ((ts < 'Sun Mar 29 17:00:00 2026 PDT'::timestamp with time zone) AND (ts >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (ts < 'Tue Mar 10 01:00:00 2026 PDT'::timestamp with time zone))
117+
Filter: ((time_bucket('@ 28 days'::interval, ts) >= 'Tue Jan 13 00:00:00 2026 PST'::timestamp with time zone) AND (time_bucket('@ 28 days'::interval, ts) < 'Tue Feb 10 00:00:00 2026 PST'::timestamp with time zone))
118+
-> HashAggregate
119+
Group Key: time_bucket('@ 7 days'::interval, (time_bucket('@ 1 day'::interval, (time_bucket('@ 6 hours'::interval, ts)))))
120+
-> Result
121+
-> HashAggregate
122+
Group Key: time_bucket('@ 1 day'::interval, (time_bucket('@ 6 hours'::interval, ts)))
123+
-> Result
124+
-> HashAggregate
125+
Group Key: time_bucket('@ 6 hours'::interval, ts)
126+
-> Result
127+
One-Time Filter: false
128+
129+
DROP TABLE conditions CASCADE;
130+
RESET client_min_messages;

0 commit comments

Comments
 (0)