After our ingesters are moved to a new cluster, we can't see the last 6 hours of data in Grafana until the next day

[jessebot]

If I move the ingesters to a new cluster or delete the PVCs, mimir comes back up just fine, and the data is all in S3, as I expect it to be, but it doesn't show up in Grafana. Is there a way to say "if data not in memory, check S3" safely?

I tried previously adjusting the limits.query_ingesters_within and limits.query_store_after values, but doing that causes inconsistent block errors. I also tried adjusting the blocks_storage.retention_period, but that also maybe causes issues.

Is there a foolproof way to delete a PVC and temporarily query S3 for short term data from the last 6 hours until the next 6 hour block?

[deniszh]

Unfortunately, Ingesters are persistent and even if you tune requests to shift load to store gateways last 2 hours of data will be lost.
So, documentation suggests to use Envoy to mirror traffic to second cluster before switching - https://grafana.com/docs/mimir/latest/configure/mirror-requests-to-a-second-cluster/