fix(monitoring): set Prometheus retention to 10y, never 0 - #97
Merged
Conversation
retention: 0d does NOT mean "keep forever". Prometheus treats retention.time=0 as unset and falls back to its built-in 15d default, so the previous change (intended as infinite retention) silently made retention far SHORTER than the 90d it replaced. Effective retention was 15d, confirmed via /api/v1/status/runtimeinfo (storageRetention: "15d"). TSDB now starts 2026-08-20; the camp period (2026-07-23..08-02) has been pruned and is not recoverable — there are no disk snapshots or backup vaults in the subscription. Use an explicit positive duration instead. retentionSize: "0" is safe to keep as a size-pruning disable now that retention.time is positive. Loki is unaffected: retention_period: 0s is genuinely "keep forever" there, and camp logs are verified intact (14.9M lines, no hourly gaps). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
The change is small, correctly scoped to the Prometheus values, and directly addresses the documented retention-time fallback behavior.
Pull request overview
This PR fixes Prometheus retention configuration in the monitoring Helm values so that retention is truly long-term instead of silently falling back to Prometheus’ default (15 days) when set to 0.
Changes:
- Set
prometheus.prometheusSpec.retentionto an explicit positive duration (10y) to avoid the0/unset fallback behavior. - Keep
prometheus.prometheusSpec.retentionSize: "0"to disable size-based pruning while time-based retention is explicitly set.
File summaries
| File | Description |
|---|---|
| k8s/infra-manifest/monitoring/helm-values.yaml | Updates Prometheus retention settings to ensure time-based retention is explicitly long-term and does not revert to the default. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
retention: 0d(merged in #96) does not mean "keep forever". Prometheus treatsretention.time=0as unset and falls back to its built-in 15 day default. The change intended as infinite retention actually made retention far shorter than the 90d it replaced.Confirmed on the running instance:
Impact
The TSDB now starts at 2026-08-20. The camp period (2026-07-23 → 08-02) has been pruned. Prometheus disk usage dropped 46.1 GB → 14.4 GB, and
prometheus_tsdb_time_retentions_total= 33 block deletions.Not recoverable —
az snapshot listandaz backup vault listare both empty for the Jamboree26 subscription.Fix
Explicit positive duration:
retentionSize: "0"is safe to keep as a size-pruning disable becauseretention.timeis now positive. The trap is specifically that both being zero/unset triggers the default.Loki is unaffected
retention_period: 0sgenuinely means keep-forever in Loki, and the compactor has deletes disabled. Camp logs verified intact: 14,924,877 lines across 2026-07-23 → 08-02, all 264 hours present, no gaps, min ~9k lines/hour.j26-prodalone has 5.85M lines over the camp peak.🤖 Generated with Claude Code