Skip to content

chore(monitoring): disable metric and log retention pruning - #96

Merged
hakan-persson merged 1 commit into
mainfrom
chore/monitoring-infinite-retention
Aug 1, 2026
Merged

chore(monitoring): disable metric and log retention pruning#96
hakan-persson merged 1 commit into
mainfrom
chore/monitoring-infinite-retention

Conversation

@hakan-persson

Copy link
Copy Markdown
Member

Keeps all metrics and logs until the camp data has been exported. The cluster is being stopped shortly and restarted later to pull data, and nothing may be pruned in the meantime.

Prometheus

retention: 90d was not the binding constraint — the size cap was:

  • 6 size-based deletions in the last 30 days, 0 time-based
  • oldest sample was only 65 days old, not 90
  • camp data was actively being deleted from the oldest end

Both are now disabled (retention: 0d, retentionSize: "0"). retentionSize is quoted so it renders as a string; a bare 0 would be an int and the Prometheus CR expects a size string.

Loki

Already kept data forever — compactor deletes are off by default and no retention_period was set anywhere in the live config. This pins both explicitly (retention_period: 0s, compactor.retention_enabled: false) so a chart-default change can't silently start pruning. No behaviour change.

Tradeoff

With no automatic pruning the Prometheus PVC no longer self-limits. Measured growth is ~0.78 GB/day against ~16.6 GB free — roughly 21 days of headroom, comfortable for the few days needed. Beyond that it becomes a hard wall, so the PVC needs watching rather than relying on a cap.

Note: the monitoring PVs are reclaimPolicy: Delete. That's fine for az aks stop/start (managed disks are preserved), but an ArgoCD prune of the monitoring app or a namespace deletion would destroy the data regardless of these settings.

🤖 Generated with Claude Code

The cluster is being stopped and restarted later to export camp data, so
nothing may be pruned in the meantime.

Prometheus was not honouring the 90d setting: the size cap was the binding
constraint, with 6 size-based deletions in the last 30 days (0 time-based)
and the oldest sample only 65 days old. Camp data was being deleted from
the oldest end. Disable both time and size retention.

Loki already kept data forever (compactor deletes are off by default and no
retention_period was set). Pin both explicitly so a chart-default change
cannot silently start pruning.

Growth is ~0.8GB/day against ~16GB free (~21 days), so with no automatic
pruning the Prometheus PVC needs watching rather than self-limiting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings August 1, 2026 05:07
@hakan-persson
hakan-persson merged commit 7961ff5 into main Aug 1, 2026
2 checks passed
@hakan-persson
hakan-persson deleted the chore/monitoring-infinite-retention branch August 1, 2026 05:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Disables automatic data pruning in the monitoring stack to ensure camp metrics and logs are retained until export, despite an upcoming cluster stop/start cycle.

Changes:

  • Prometheus: disables both time- and size-based TSDB retention to prevent any pruning.
  • Loki: explicitly pins infinite retention and disables compactor retention deletes to guard against future chart default changes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
k8s/infra-manifest/monitoring/helm-values.yaml Sets Prometheus retention and retentionSize to disable pruning, with updated operational notes.
k8s/infra-manifest/monitoring/loki-values.yaml Explicitly sets Loki retention_period: 0s and compactor.retention_enabled: false to prevent retention from being enabled via defaults.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants