Cap refresh start when hypertable has tiered data - #9811
Open
kpan2034 wants to merge 5 commits into
Open
Conversation
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
May 27, 2026 14:28
0ef8281 to
3e5c07d
Compare
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
June 5, 2026 20:33
3e5c07d to
221041a
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
June 5, 2026 21:13
221041a to
fbd5fcd
Compare
kpan2034
marked this pull request as ready for review
June 5, 2026 21:13
|
@melihmutlu, @natalya-aksman: please review this pull request.
|
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
June 5, 2026 21:20
fbd5fcd to
1ed30b3
Compare
kpan2034
changed the title
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
June 5, 2026 21:23
1ed30b3 to
3ddfbf5
Compare
melihmutlu
reviewed
Jun 8, 2026
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
4 times, most recently
from
June 10, 2026 23:43
6d40793 to
ff7856d
Compare
melihmutlu
reviewed
Jun 15, 2026
Currently during a refresh, we process the ranges before the earliest chunk in the hypertable as well. This can lead to potential missing data when tiered data is present but tiered reads are disabled during the refresh. By capping the refresh range at the start of the earliest chunk, any invalidations before the earliest chunk are no longer processed and removed. Thus, in a subsequent refresh, if tiered data reads are enabled, that data would be materialized in the CAgg. We only need to do this when the hypertable has tiered data. Any new tiered data will either have been processed before tiering (since it exists in some chunk) or will be inserted after this refresh (which will generate invalidations).
Capping is done incorrectly when the OSM chunk is the earliest chunk.
Incremental batch generation would generate batches considering tiered data ranges, even if tiered reads are disabled. This can lead to invalid batches since we cap the refresh start to the earliest chunk in the hypertable during the refresh.
kpan2034
force-pushed
the
cap-invals-at-chunk-min
branch
from
July 1, 2026 15:46
71d94c6 to
321f063
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently during a refresh, we process the ranges before the earliest chunk in the hypertable as well. This can lead to potential missing data when tiered data is present but tiered reads are disabled during the refresh.
By capping the refresh range at the start of the earliest chunk, any invalidations before the earliest chunk are no longer processed and removed. Thus, in a subsequent refresh, if tiered data reads are enabled, that data would be materialized in the CAgg.
We only need to do this when the hypertable has tiered data. Any new tiered data will either have been processed before tiering (since it exists in some chunk) or will be inserted after this refresh (which will generate invalidations).
We also cap the refresh start when generating batches for an incremental refresh.