Fix flaky cagg_hierarchical_concurrent_refresh - #10349
Conversation
|
@dbeck, @svenklemm: please review this pull request.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
e582aa4 to
7547859
Compare
9a083dd to
1cdd927
Compare
| step unlock_6h_mat_table: | ||
| ROLLBACK; | ||
|
|
||
| <waiting ...> |
There was a problem hiding this comment.
Do you know why we see waiting here?
There was a problem hiding this comment.
Because it waits for a previous refresh to complete, "unlock_6h_mat_table"("L2_refresh_full"). Maybe that's not necessary.
There was a problem hiding this comment.
That dependency just makes sure that unlock happens only after L2_refresh_full starts, right?
There was a problem hiding this comment.
It was to have the same output order in every run. But that wasn't needed for unlock_6h_mat_table, so I removed it.
1cdd927 to
87af8e0
Compare
| cagg_1d|Mon Jan 05 00:00:00 2026 UTC|infinity | ||
| cagg_6h|-infinity |Wed Dec 31 23:59:59.999999 2025 UTC | ||
| cagg_6h|Fri Jan 02 00:00:00 2026 UTC|Sat Jan 03 23:59:59.999999 2026 UTC | ||
| cagg_6h|Fri Jan 02 00:00:00 2026 UTC|Fri Jan 02 23:59:59.999999 2026 UTC |
There was a problem hiding this comment.
see this diff. I believe this is executing in batches and can cause non-deterministic behavior again. Could you check if we need to force setting as num_batches=0
There was a problem hiding this comment.
Yes, it runs in batches. But the diff is because there is an additional refresh in the beginning (L1b_refresh_jan3) to generate some invalidations for L2 refresh. So Jan 3 is already refreshed which wasn't the case previously.
| # Once L1 is blocked in txn3 and still holding its lock on the materialization invalidation | ||
| # log, since it hasn't committed yet, run L2's refresh. L2 should get through txn1 and txn2 | ||
| # without any issue despite that lock. Once L2 also hits the block in txn3, release it. | ||
| # |
There was a problem hiding this comment.
I think some more comments are helpful here:
L1_refresh_jan1 refreshes Jan1. But is blocked in Txn3 and so changes are not yet materialized when L2_refresh_full is in Txn2. So Jan1 changes are not visible to L2 and it only updates data for Jan3.
87af8e0 to
862548f
Compare
gayyappan
left a comment
gayyappan
left a comment
There was a problem hiding this comment.
request for an additional comment
Synchronize with a lock on cagg_6h's materialization hypertable instead. L1 cagg refresh is blocked on the lock mid-txn 3 after deleting its processed log entries but before the commit. While L2 moves and process its log entries concurrently, blocking only when it starts materializing from cagg_6h.
862548f to
5688c6b
Compare
melihmutlu
added
the
force-auto-backport
Synchronize with a lock on cagg_6h's materialization hypertable instead. L1 cagg refresh is blocked on the lock mid-txn 3 after deleting its processed log entries but before the commit. While L2 moves and process its log entries concurrently, blocking only when it starts materializing from cagg_6h. (cherry picked from commit 62575c3)
4 participants
Synchronize with a lock on cagg_6h's materialization hypertable instead. L1 cagg refresh is blocked on the lock mid-txn 3 after deleting its processed log entries but before the commit. While L2 moves and process its log entries concurrently, blocking only when it starts materializing from cagg_6h.