chore(monitoring): raise Prometheus retentionSize 50GB->60GB - #95
Merged
Conversation
The 50GB size cap was the binding retention limit (below retention: 90d), so Prometheus was pruning the oldest data early — TSDB only held ~61 days instead of 90. During camp week the higher ingest would keep advancing that deletion front toward the present, risking loss of early-camp data needed for evaluation. Raise the cap to 60GB (PVC is ~46GB used of ~67GB usable) so time-based 90d retention becomes the real limit, keeping ~7GB raw-disk headroom for WAL/head/compaction. No pod restart or disk resize needed. A proper PVC grow (and moving Prometheus to the D8as_v5 pool for online resize) is a separate follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR adjusts the kube-prometheus-stack configuration in this repo’s monitoring Helm values to increase Prometheus TSDB size-based retention, so the configured 90d time retention is less likely to be cut short by the size cap.
Changes:
- Increased Prometheus
retentionSizefrom50GBto60GB. - Added inline documentation explaining the rationale and PVC headroom considerations.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Raises Prometheus
retentionSizefrom50GBto60GBin the monitoring Helm values.Why
retention: 90dandretentionSize: 50GBwere both set; Prometheus enforces whichever is hit first. The 50GB size cap was the binding limit — the TSDB only held ~61 days of data instead of 90, i.e. it was already pruning the oldest data early. During camp week (higher ingest + new Traefik metrics) that deletion front would keep advancing toward the present, risking loss of early-camp data needed for the evaluation.Raising the cap to 60GB (PVC is ~46GB used of ~67GB usable) makes 90d time retention the real limit again, keeping ~7GB raw-disk headroom for WAL/head/compaction.
Safety / rollout
Follow-up
🤖 Generated with Claude Code