[Guidance] Thanos store max-time and consistency-delay

[realRunlo]

Hi everyone!

This is the setup i'm working on:

flowchart LR

subgraph LandingZone["Landing Zone"]

    %% Azure Cloud
    subgraph Azure["Azure Cloud"]
        subgraph AzureScrape["Scrape"]
            AZA[Prometheus Agent]
        end
    end

    %% AWS Cloud
    subgraph AWS["AWS Cloud"]

        subgraph Scrape["Scrape"]
            A[Prometheus Agent / ADOT Collector]
        end

        subgraph Ingestion["Ingestion"]
            B[Receiver Router]
            C[Receiver Ingesting]
            A -- Remote write --> B --> C
        end

        subgraph Storage["Storage"]
            D[🪣 S3]
            E[Storeger]
            G[Compactor]
            D --> E
            G --> D
        end

        subgraph Query["Query"]
            F[Querier]
        end

        C --> D
        F -- Store API --> E
        F -- Store API --> C
    end

    %% Cross-cloud connection
    AZA -- Remote write --> B

end

Loading

All components run on ECS using Spot Instances. The receiver-ingester runs without persistent storage, and TSDB blocks are flushed to S3 every two hours or when a task terminates.

I am running without persistent storage for two reasons:

• Attaching an EBS volume to a service means that tasks are tied to a specific Availability Zone.
• Attaching an EFS volume would likely increase latency.

My question concerns the configuration of max-time and consistency-delay. In my setup, there is a possibility that, at a given time, only S3 contains samples to serve. This can happen because Spot Instance rotation causes all tasks that contain a given serie to terminate, or because I redeploy the receiver-ingesting to update resource allocation.

If I configure max-time to 24 hours, I will have a gap in metrics for the data that was flushed by the receiver-ingesting for up to 24 hours. The same logic applies to the consistency-delay configuration.

My only alternative is to not set max-time and to set consistency-delay to a very low value, for example, 15 seconds.

Is this setup recommended? What can go wrong, aside from increasing S3 operations and therefore costs?

Thanks in advance for your input.