feat: Add time-window conversion via min/max TSDB offsets (day-rounded)#31
feat: Add time-window conversion via min/max TSDB offsets (day-rounded)#31TaranpreetSingh wants to merge 15 commits intothanos-io:mainfrom
Conversation
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
089edd4 to
723fe50
Compare
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
ae7d0ce to
f2cc0da
Compare
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
| } | ||
| log.Info("Converting next blocks", "sort_by", opts.conversion.sortLabels) | ||
| if err := advanceConversion(iterCtx, log, tsdbBkt, parquetBkt, tsdbDiscoverer, parquetDiscoverer, opts.conversion); err != nil { | ||
| if err := advanceConversion(iterCtx, log, tsdbBkt, parquetBkt, tsdbDiscoverer, parquetDiscoverer, opts.conversion, opts.conversion.minTimeOffset, opts.conversion.maxTimeOffset); err != nil { |
There was a problem hiding this comment.
they are already on conversion opts, we dont need to pass explicitely.
There was a problem hiding this comment.
fixed, by removing them for the func parameter and calling them directly in plan
|
Generally looks good to me, but can you address the nit above and maybe also clean up some of the comments and address the commented test issue? |
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
When we added the time based sharding , it was not clear how should be passed the real time time range / set the time range for the test block in real time, as the plan use the time.now() cal for the time based sharding. This is been fixed now |
Signed-off-by: Taranpreet Singh <TaranpreetSingh@users.noreply.github.com>
rebase with main changes Signed-off-by: Taranpreet Singh <TaranpreetSingh@users.noreply.github.com>
rebases with the main Signed-off-by: Taranpreet Singh <TaranpreetSingh@users.noreply.github.com>
Signed-off-by: Taranpreet Singh <TaranpreetSingh@users.noreply.github.com>
GiedriusS
left a comment
There was a problem hiding this comment.
Why not just specify everything in days if everything is already rounded to a day? I'm trying to wrap my head around the problem this solves. Is the goal to run multiple converters for a different subset of inputs by day?
Yes, as it can help in multiple ways, like in case of backfill. |
Signed-off-by: Taranpreet Singh <tsingh@roku.com>
Signed-off-by: Taranpreet Singh <TaranpreetSingh@users.noreply.github.com>
| winStartDay = util.NewDate(t.Year(), t.Month(), t.Day()).ToTime() | ||
| // Parse minDate and round to start of day (00:00:00 UTC, inclusive) | ||
| if minTimeDate != "" { | ||
| t, err := time.Parse("2006-01-02", minTimeDate) |
There was a problem hiding this comment.
Same constant 2006-01-02 could be reused. Also, could we move validation one layer up - the caller of this function could check if the inputs are correct and just pass time.Time, no? If a non-empty string is passed then I would expect it to be checked and error out if it doesn't follow the format that is expected.
Introduce min/max TimeDate to support for conversion planning using convert.min-time and convert.max-time.
This change enables day-rounded time window selection for conversion via min/max time range. It improves operational control by focusing conversions on a targeted period, reducing unnecessary downloads, compute, and storage for out-of-window data. It supports backfill and selective reprocessing scenarios while preserving default behavior when duration are unset. Duration are interpreted as explicit date and rounded to day boundaries (min to 00:00:00 UTC; max to next day start, exclusive), ensuring consistent, predictable daily block generation.
Key changes
Test result
with flag used