fix(monitoring): stop Loki high-memory false alerts (cgroup-v1 slab)

[alex-struk]

Problem

The recurring HighPodMemoryUsage critical alert on bcgov-di-plg-loki-0 (fd34fb-prod) is a false alarm, and prior attempts (three memory-limit bumps + a chunk_idle_period cut) never resolved it.

Live diagnosis on the prod pod (via oc exec into its cgroup):

Metric	Value
memory.usage_in_bytes	1.74 GiB / 2 GiB limit (peaked 1.98 GiB)
memory.kmem.usage_in_bytes (kernel slab)	1.59 GiB
rss (Loki's actual Go heap)	~99 MiB
Restarts	0 over ~6 days
Node	RHEL 9.4, kernel 5.14, cgroup v1

Root cause: the alert uses container_memory_working_set_bytes / limit, which on cgroup v1 includes reclaimable kernel slab (dentry/inode caches). Loki's continuous chunk create/delete churn (ingester flush + compaction + retention deletes) inflates that slab until it nears the 2 GiB limit. Loki's real footprint is ~100 MiB; the kernel reclaims the slab before any OOM — which is why the pod sits near 90% forever with zero restarts.

Why the prior fixes failed:

• Memory-limit bumps → slab just refills the larger limit. Can't work.
• chunk_idle_period 30m→5m → made it worse (more frequent flushes = more files = more dentry churn).

Corroboration: Red Hat solution 7126558 (xfs_inode/dentry slab OOM under cgroup v1); Grafana docs discourage Loki on shared/NFS filesystem stores.

Changes

1. prometheus-rule-crd.yaml — exclude the Loki pod from HighPodMemoryUsage (working-set is a meaningless signal for it under cgroup v1). The exclusion is derived from the plg.loki.fullname helper. All other pods keep the early-warning alert. A genuine Loki OOM is still caught by the existing PodInErrorState (OOMKilled) rule.
2. loki-configmap.yaml — chunk_idle_period 5m→1h, add explicit max_chunk_age: 2h, chunk_target_size 1.5Mi→2Mi. Fewer, larger chunk files → less dentry/inode slab churn at the source. Heap headroom makes the longer idle period safe.
3. docs-md/ALERTING.md — documents the infra-alert template location and the cgroup-v1 exclusion rationale.

Validation

• helm lint passes; helm template (release bcgov-di-plg, ns fd34fb-prod) renders the exclusion as pod!~"bcgov-di-plg-loki-.*" and the ingester block as chunk_idle_period: 1h / max_chunk_age: 2h / chunk_target_size: 2097152.
• No backend code touched (no unit tests apply).

Not included (deliberately)

• Memory-limit changes (leaving 2 Gi).
• Retention reduction (product tradeoff — excluded per request).
• Storage-class move (NFS → block) and cgroup-v2 node migration — the structural cure — tracked as follow-ups.

Rollout note

Applies on next PLG helm upgrade. The already-firing alert instance can be silenced in Alertmanager in the meantime (it's not a real risk — 0 restarts, ~99 MiB heap).

🤖 Generated with Claude Code

[kmandryk]

I believe this may have already been fixed, however prod deployments are currently broken. The fix would be in #222, but the latest deployment to main that was successful was #217.