Skip to content

Backport to 2.29.x: #10349: Fix flaky cagg_hierarchical_concurrent_refresh #10375

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
timescale-automation merged 2 commits into from
Jul 31, 2026
+71 −58
Merged

Backport to 2.29.x: #10349: Fix flaky cagg_hierarchical_concurrent_refresh #10375

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8496a62
Fix flaky cagg_hierarchical_concurrent_refresh (#10349)
melihmutlu Jul 31, 2026
42c3b8a
Merge remote-tracking branch 'origin/2.29.x' into HEAD
timescale-automation Jul 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 35 additions & 38 deletions tsl/test/isolation/expected/cagg_hierarchical_concurrent_refresh.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Parsed test spec with 8 sessions
Parsed test spec with 7 sessions

starting permutation: chk_hyper_invals WP_before_enable L1_refresh_jan1 WP_before_release chk_hyper_invals WP_before_enable L2_refresh_jan1 WP_before_release chk_cagg_6h chk_cagg_1d chk_6h_consistency chk_1d_consistency
step chk_hyper_invals:
Expand Down Expand Up @@ -463,7 +463,7 @@ Fri Jan 02 00:00:00 2026 UTC|32.1666666666667|32.1666666666667| 288|
Sat Jan 03 00:00:00 2026 UTC| 31.5| 31.5| 96| 96|t |t


starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2_refresh_jan1 insert_ht L1_refresh_full L2b_refresh_jan1_2 WP_before_release unlock chk_cagg_1d chk_1d_consistency
starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2_refresh_jan1 insert_ht L1_refresh_full L2b_refresh_jan1_2 WP_before_release unlock_L2_source chk_cagg_1d chk_1d_consistency
step L1_refresh_full:
CALL refresh_continuous_aggregate('cagg_6h', '2026-01-01', '2026-01-04');

Expand Down Expand Up @@ -546,7 +546,7 @@ debug_waitpoint_release


step L1_refresh_full: <... completed>
step unlock:
step unlock_L2_source:
ROLLBACK;

step L2_refresh_jan1: <... completed>
Expand Down Expand Up @@ -586,7 +586,7 @@ Fri Jan 02 00:00:00 2026 UTC| 31.5|32.1666666666667| 96|
Sat Jan 03 00:00:00 2026 UTC| 31.5| 40.5| 96| 288|f |f


starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2b_refresh_jan1_2 insert_ht L1_refresh_full L2_refresh_jan1 WP_before_release unlock chk_cagg_1d chk_1d_consistency
starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2b_refresh_jan1_2 insert_ht L1_refresh_full L2_refresh_jan1 WP_before_release unlock_L2_source chk_cagg_1d chk_1d_consistency
step L1_refresh_full:
CALL refresh_continuous_aggregate('cagg_6h', '2026-01-01', '2026-01-04');

Expand Down Expand Up @@ -669,7 +669,7 @@ debug_waitpoint_release


step L1_refresh_full: <... completed>
step unlock:
step unlock_L2_source:
ROLLBACK;

step L2b_refresh_jan1_2: <... completed>
Expand Down Expand Up @@ -864,7 +864,7 @@ Fri Jan 02 00:00:00 2026 UTC| 31.5| 31.5| 96|
Sat Jan 03 00:00:00 2026 UTC| 40.5| 40.5| 288| 288|t |t


starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2_refresh_jan1 insert_ht L1_refresh_full L2b_refresh_jan3 WP_before_release unlock chk_cagg_1d chk_1d_consistency
starting permutation: L1_refresh_full chk_1d_consistency WP_before_enable chk_hyper_invals lock_L2_source L2_refresh_jan1 insert_ht L1_refresh_full L2b_refresh_jan3 WP_before_release unlock_L2_source chk_cagg_1d chk_1d_consistency
step L1_refresh_full:
CALL refresh_continuous_aggregate('cagg_6h', '2026-01-01', '2026-01-04');

Expand Down Expand Up @@ -946,7 +946,7 @@ debug_waitpoint_release


step L1_refresh_full: <... completed>
step unlock:
step unlock_L2_source:
ROLLBACK;

step L2_refresh_jan1: <... completed>
Expand Down Expand Up @@ -987,19 +987,15 @@ Fri Jan 02 00:00:00 2026 UTC| 31.5|32.1666666666667| 96|
Sat Jan 03 00:00:00 2026 UTC| 40.5| 40.5| 288| 288|t |t


starting permutation: insert_ht WP_after_enable chk_6h_consistency L1_refresh_jan1 lock_mat_invals WP_after_release L2_refresh_full unlock chk_mat_invals chk_6h_consistency chk_1d_consistency
starting permutation: insert_ht L1b_refresh_jan3 chk_6h_consistency lock_6h_mat_table L1_refresh_jan1 L2_refresh_full unlock_6h_mat_table chk_mat_invals chk_6h_consistency chk_1d_consistency
step insert_ht:
INSERT INTO conditions (time, temp)
SELECT ts, 50.0
FROM generate_series('2026-01-01 00:00:00+00'::timestamptz,
'2026-01-04 23:45:00+00', '15 minutes') ts;

step WP_after_enable:
SELECT debug_waitpoint_enable('after_process_cagg_invalidations_for_refresh_lock');

debug_waitpoint_enable
----------------------

step L1b_refresh_jan3:
CALL refresh_continuous_aggregate('cagg_6h', '2026-01-03 00:00', '2026-01-04 00:00');

step chk_6h_consistency:
-- Verify L1 (6-hour) matches re-aggregation from raw hypertable
Expand Down Expand Up @@ -1029,34 +1025,35 @@ Fri Jan 02 00:00:00 2026 UTC| 22.5| 27.5| 24| 72|f
Fri Jan 02 06:00:00 2026 UTC| 28.5| 29.5| 24| 72|f |f
Fri Jan 02 12:00:00 2026 UTC| 34.5|34.8333333333333| 24| 72|f |f
Fri Jan 02 18:00:00 2026 UTC| 40.5|36.8333333333333| 24| 72|f |f
Sat Jan 03 00:00:00 2026 UTC| 22.5| 37.5| 24| 72|f |f
Sat Jan 03 06:00:00 2026 UTC| 28.5| 39.5| 24| 72|f |f
Sat Jan 03 12:00:00 2026 UTC| 34.5| 41.5| 24| 72|f |f
Sat Jan 03 18:00:00 2026 UTC| 40.5| 43.5| 24| 72|f |f
Sat Jan 03 00:00:00 2026 UTC| 37.5| 37.5| 72| 72|t |t
Sat Jan 03 06:00:00 2026 UTC| 39.5| 39.5| 72| 72|t |t
Sat Jan 03 12:00:00 2026 UTC| 41.5| 41.5| 72| 72|t |t
Sat Jan 03 18:00:00 2026 UTC| 43.5| 43.5| 72| 72|t |t

step lock_6h_mat_table:
BEGIN;
DO $$
DECLARE
mat_table text;
BEGIN
SELECT format('%I.%I', h.schema_name, h.table_name) INTO mat_table
FROM _timescaledb_catalog.continuous_agg ca
JOIN _timescaledb_catalog.hypertable h ON h.id = ca.mat_hypertable_id
WHERE ca.user_view_name = 'cagg_6h';
EXECUTE format('LOCK TABLE %s IN ACCESS EXCLUSIVE MODE', mat_table);
END;
$$;

step L1_refresh_jan1:
CALL refresh_continuous_aggregate('cagg_6h', '2026-01-01 00:00', '2026-01-02 00:00');
<waiting ...>
step lock_mat_invals:
BEGIN;
LOCK _timescaledb_catalog.continuous_aggs_materialization_invalidation_log
IN ACCESS EXCLUSIVE MODE;

step WP_after_release:
SELECT debug_waitpoint_release('after_process_cagg_invalidations_for_refresh_lock');

debug_waitpoint_release
-----------------------


step L2_refresh_full:
CALL refresh_continuous_aggregate('cagg_1d', '2026-01-01', '2026-01-05');
<waiting ...>
step unlock:
step unlock_6h_mat_table:
ROLLBACK;

step L1_refresh_jan1: <... completed>
L2: NOTICE: continuous aggregate "cagg_1d" is already up-to-date
step L2_refresh_full: <... completed>
step chk_mat_invals:
SELECT ca.user_view_name AS cagg,
Expand All @@ -1071,7 +1068,7 @@ cagg |lowest |greatest
cagg_1d|-infinity |Wed Dec 31 23:59:59.999999 2025 UTC
cagg_1d|Mon Jan 05 00:00:00 2026 UTC|infinity
cagg_6h|-infinity |Wed Dec 31 23:59:59.999999 2025 UTC
cagg_6h|Fri Jan 02 00:00:00 2026 UTC|Sat Jan 03 23:59:59.999999 2026 UTC
cagg_6h|Fri Jan 02 00:00:00 2026 UTC|Fri Jan 02 23:59:59.999999 2026 UTC
cagg_6h|Sun Jan 04 00:00:00 2026 UTC|infinity

step chk_6h_consistency:
Expand Down Expand Up @@ -1102,10 +1099,10 @@ Fri Jan 02 00:00:00 2026 UTC| 22.5| 27.5| 24|
Fri Jan 02 06:00:00 2026 UTC| 28.5| 29.5| 24| 72|f |f
Fri Jan 02 12:00:00 2026 UTC| 34.5|34.8333333333333| 24| 72|f |f
Fri Jan 02 18:00:00 2026 UTC| 40.5|36.8333333333333| 24| 72|f |f
Sat Jan 03 00:00:00 2026 UTC| 22.5| 37.5| 24| 72|f |f
Sat Jan 03 06:00:00 2026 UTC| 28.5| 39.5| 24| 72|f |f
Sat Jan 03 12:00:00 2026 UTC| 34.5| 41.5| 24| 72|f |f
Sat Jan 03 18:00:00 2026 UTC| 40.5| 43.5| 24| 72|f |f
Sat Jan 03 00:00:00 2026 UTC| 37.5| 37.5| 72| 72|t |t
Sat Jan 03 06:00:00 2026 UTC| 39.5| 39.5| 72| 72|t |t
Sat Jan 03 12:00:00 2026 UTC| 41.5| 41.5| 72| 72|t |t
Sat Jan 03 18:00:00 2026 UTC| 43.5| 43.5| 72| 72|t |t

step chk_1d_consistency:
-- Verify L2 (daily) matches re-aggregation from L1 (6-hour)
Expand All @@ -1129,5 +1126,5 @@ bucket |daily_avg| from_6h_avg|daily_count|from_6h_coun
----------------------------+---------+----------------+-----------+-------------+---------+-----------
Thu Jan 01 00:00:00 2026 UTC| 31.5|37.1666666666667| 96| 288|f |f
Fri Jan 02 00:00:00 2026 UTC| 31.5| 31.5| 96| 96|t |t
Sat Jan 03 00:00:00 2026 UTC| 31.5| 31.5| 96| 96|t |t
Sat Jan 03 00:00:00 2026 UTC| 40.5| 40.5| 288| 288|t |t

56 changes: 36 additions & 20 deletions tsl/test/isolation/specs/cagg_hierarchical_concurrent_refresh.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,16 +116,6 @@ step "WP_before_release"
SELECT debug_waitpoint_release('before_process_cagg_invalidations_for_refresh_lock');
}

session "WP_after"
step "WP_after_enable"
{
SELECT debug_waitpoint_enable('after_process_cagg_invalidations_for_refresh_lock');
}
step "WP_after_release"
{
SELECT debug_waitpoint_release('after_process_cagg_invalidations_for_refresh_lock');
}

# Session to refresh L1 (6-hour)
session "L1"
setup
Expand Down Expand Up @@ -210,13 +200,27 @@ step "lock_L2_source"
END;
$$;
}
step "lock_mat_invals"
step "unlock_L2_source"
{
ROLLBACK;
}

step "lock_6h_mat_table"
{
BEGIN;
LOCK _timescaledb_catalog.continuous_aggs_materialization_invalidation_log
IN ACCESS EXCLUSIVE MODE;
DO $$
DECLARE
mat_table text;
BEGIN
SELECT format('%I.%I', h.schema_name, h.table_name) INTO mat_table
FROM _timescaledb_catalog.continuous_agg ca
JOIN _timescaledb_catalog.hypertable h ON h.id = ca.mat_hypertable_id
WHERE ca.user_view_name = 'cagg_6h';
EXECUTE format('LOCK TABLE %s IN ACCESS EXCLUSIVE MODE', mat_table);
END;
$$;
}
step "unlock"
step "unlock_6h_mat_table"
{
ROLLBACK;
}
Expand Down Expand Up @@ -317,10 +321,10 @@ permutation "WP_before_enable" "chk_hyper_invals" "L1b_refresh_jan1_2" "insert_h

# Two concurrent refreshes on L2 CAgg with overlapping ranges. One should fail due to overlap.
# L1 is refreshed between the two L2 refreshes so that the second L2 refresh encounters an overlapping materialization range and fails.
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2_refresh_jan1" "insert_ht" "L1_refresh_full" "L2b_refresh_jan1_2" "WP_before_release" "unlock" "chk_cagg_1d" "chk_1d_consistency"
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2_refresh_jan1" "insert_ht" "L1_refresh_full" "L2b_refresh_jan1_2" "WP_before_release" "unlock_L2_source" "chk_cagg_1d" "chk_1d_consistency"

# Same as above, reverse refresh order.
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2b_refresh_jan1_2" "insert_ht" "L1_refresh_full" "L2_refresh_jan1" "WP_before_release" "unlock" "chk_cagg_1d" "chk_1d_consistency"
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2b_refresh_jan1_2" "insert_ht" "L1_refresh_full" "L2_refresh_jan1" "WP_before_release" "unlock_L2_source" "chk_cagg_1d" "chk_1d_consistency"

# Non-overlapping concurrent refreshes on L1. Both refreshes should succeed. Both are blocked before Txn 3.
# Refreshing Jan 3 adds cagg invalidations that are partially overlapping with the one created by refreshing Jan 1.
Expand All @@ -330,8 +334,20 @@ permutation "WP_before_enable" "chk_hyper_invals" "L1_refresh_jan1" "insert_ht"
# Non-overlapping concurrent refresh on L2. Both refreshes should succeed. Both are blocked before Txn 3.
# Refreshing Jan 3 adds cagg invalidations that are partially overlapping with the one created by refreshing Jan 1.
# Those invalidations are left behind, but both Jan 1 and Jan 3 are refreshed successfully.
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2_refresh_jan1" "insert_ht" "L1_refresh_full" "L2b_refresh_jan3" "WP_before_release" "unlock" "chk_cagg_1d" "chk_1d_consistency"
permutation "L1_refresh_full" "chk_1d_consistency" "WP_before_enable" "chk_hyper_invals" "lock_L2_source" "L2_refresh_jan1" "insert_ht" "L1_refresh_full" "L2b_refresh_jan3" "WP_before_release" "unlock_L2_source" "chk_cagg_1d" "chk_1d_consistency"

# L1 (txn3, deleting/inserting mat_inval entries) and L2 (txn2, processing mat_inval entries) should not block each other.
# Lock materialization invalidation table to make both refreshes wait before processing entries, then release simultaneously.
permutation "insert_ht" "WP_after_enable" "chk_6h_consistency" "L1_refresh_jan1" "lock_mat_invals" "WP_after_release" "L2_refresh_full" "unlock" "chk_mat_invals" "chk_6h_consistency" "chk_1d_consistency"
# Lock cagg_6h's materialization table to block L1's refresh in txn3, before it makes any
# change to that table. Any change there generates invalidations for L2's cagg (cagg_1d),
# so we synchronize both refreshes around it.
#
# Once L1 is blocked in txn3 and still holding its lock on the materialization invalidation
# log, since it hasn't committed yet, run L2's refresh. L2 should get through txn1 and txn2
# without any issue despite that lock. Once L2 also hits the block in txn3, release it.
#
# L1_refresh_jan1 refreshes Jan 1, but is blocked in txn3, so its changes aren't materialized
# yet while L2_refresh_full processes invalidations in txn1 and txn2. Jan 1 is therefore not
# visible to L2 and it only updates data for Jan 3.
#
# There shouldn't be a race condition on materialization invalidation reads/writes between
# the two refreshes, so both refreshes should succeed.
permutation "insert_ht" "L1b_refresh_jan3" "chk_6h_consistency" "lock_6h_mat_table" "L1_refresh_jan1" "L2_refresh_full"("L1_refresh_jan1") "unlock_6h_mat_table" "chk_mat_invals" "chk_6h_consistency" "chk_1d_consistency"
Loading