Skip to content

Commit 7961ff5

Browse files
hakan-perssonclaude
andcommitted
chore(monitoring): disable metric and log retention pruning
The cluster is being stopped and restarted later to export camp data, so nothing may be pruned in the meantime. Prometheus was not honouring the 90d setting: the size cap was the binding constraint, with 6 size-based deletions in the last 30 days (0 time-based) and the oldest sample only 65 days old. Camp data was being deleted from the oldest end. Disable both time and size retention. Loki already kept data forever (compactor deletes are off by default and no retention_period was set). Pin both explicitly so a chart-default change cannot silently start pruning. Growth is ~0.8GB/day against ~16GB free (~21 days), so with no automatic pruning the Prometheus PVC needs watching rather than self-limiting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 072b891 commit 7961ff5

2 files changed

Lines changed: 10 additions & 5 deletions

File tree

k8s/infra-manifest/monitoring/helm-values.yaml

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -86,11 +86,11 @@ prometheus:
8686
operator: In
8787
values:
8888
- agentpool
89-
retention: 90d
90-
# Size cap must stay below the PVC's usable disk (~67GB of the 64Gi request)
91-
# with headroom for WAL/head/compaction. Raised from 50GB (which was pruning
92-
# oldest data before 90d) so camp+evaluation data is retained by time, not size.
93-
retentionSize: 60GB
89+
# Retention fully disabled (0 = never prune, by time or size): camp data must
90+
# survive until it has been exported. Growth is ~0.8GB/day against ~16GB free,
91+
# so the PVC must be watched rather than relying on automatic pruning.
92+
retention: 0d
93+
retentionSize: "0"
9494
storageSpec:
9595
volumeClaimTemplate:
9696
spec:

k8s/infra-manifest/monitoring/loki-values.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ loki:
2424
auth_enabled: false
2525
limits_config:
2626
allow_structured_metadata: false
27+
# 0s = keep forever. Explicit so a chart-default change can't start pruning
28+
# camp logs before they are exported.
29+
retention_period: 0s
30+
compactor:
31+
retention_enabled: false
2732

2833
singleBinary:
2934
replicas: 1

0 commit comments

Comments
 (0)