: No data after restart (monolithic Mimir, using S3 backend)

[TatevHunanyan-BM]

Hi, I’m running Mimir in monolithic mode on AWS ECS and using S3 as the backend storage. I’m having an issue where:

Metrics are ingested correctly

I can see TSDB blocks stored in S3 under the correct tenant prefix

But after restarting the ECS task, Grafana shows no data, even though the blocks are present in S3.

server:
http_listen_port: 9091
grpc_listen_port: 9009
log_level: info
log_format: logfmt

distributor:
ring:
kvstore:
store: memberlist

ingester:
ring:
kvstore:
store: memberlist
replication_factor: 1

common:
storage:
backend: s3
s3:
region: us-west-1
endpoint: s3.us-west-1.amazonaws.com
bucket_name: dev-monitoring

blocks_storage:
backend: s3
tsdb:
dir: /var/lib/mimir/data/ingester
flush_blocks_on_shutdown: true
bucket_store:
sync_dir: /var/lib/mimir/data/tsdb-sync/

ruler_storage:
backend: filesystem

alertmanager_storage:
backend: filesystem

[deniszh]

Mimir caching last 12 hours of data in ingesters, so /var/lib/mimir/data/ingester data in your case should be persistent

[TatevHunanyan-BM]

@deniszh I want to configure machines in AWS ECS forgate, but I can't quite figure out if I can use one EFS storage that will be mounted to all instances?
Or maybe I should use a separate EFS for each instance?

If it is a separate EFS, can I be sure that if the instances are terminated correctly during scaling, all the data from the EFS will go to s3?

[deniszh]

@TatevHunanyan-BM : sorry, ECS is not my cup of tea, only worked with EFS in K8S, it worked although much more expensive than EBS, so, probably not relevant to your use case.