A strange "gap" in my Thanos metrics viewed in Grafana (timezone related, I believe)

[gareth-e]

Hi,
I have a strange gap showing in metrics in Grafana which I believe is time-zone related (I'm in UK, so now in daylight-saving BST.... UTC+1).

This is what I see:

On the left is a metric (blackbox exporter probe_icmp_duration_seconds) coming from a Thanos (Query Frontend) datasource and on the right is the same metric direct from the Prometheus datasource.

Regarding the Thanos timeseries: before the gap is coming from Thanos Store - and after is coming from Sidecar (Query is reading from both for a complete picture - or so I'd hoped).
I have Sidecar configured with --min-time=-120m so it's not returning metrics that are also coming from Store (to save on deduplication effort, and reduce network overhead - a limitation here). I see this is working... metrics from 11:50 BST to 13:50 BST are visible. These timeseries were observed at 13:50 BST.
But Store metrics appear to stop at 11:04 BST.
I don't believe this was happening before clocks changed (I didn't see this when we were in GMT... UTC+0).

I can see Sidecar uploading OK every 2 hours without issues:

(this is showing in BST)

...and in the logs (these are UTC) all seems fine:

I guess I could "fix" this by just getting Sidecar to return the last 3 hours, not 2 - but I'd really like to understand what's going on.

Sidecar uploaded at 12:00 BST but it looks like it only uploaded (or Store is only returning) up to 11:00 BST. But that doesn't sound right. My reasoning why it's 11:00 BST is by comparing the peaks on the left and right timeseries - I can see there is nothing missing and all up to current time on Prometheus - and the peaks in Thanos metrics are in sync. So this must be (correctly) in BST.

Maybe I'm going mad here or just not seeing it. But why would Thanos Store only return metrics up to 11:00 BST when a Sidecar upload happened at 12:00 BST?

Has anyone else experienced this or similar?

Or have any ideas on how to troubleshoot in a deeper way?

Many thanks.

[gareth-e]

I'm wondering if this might be more related to Prometheus TSDB behaviour... what I notice in Prometheus is that the Blocks / head is never fully up-to-date:

The Prometheus TSDB block creation (every 2 hours, as per this Sidecar config... min-block-duration and max-block-duration both 2 hours) results in Prometheus Block (and then Thanos Store) data being between 1 and 3 hours old (not zero and 2).

So I'm curious if this is "normal" behaviour? Or should the block creation bring TSDB blocks up to current time? 🤔