fix(monitoring): stop Loki high-memory false alerts from cgroup-v1 slab

The recurring HighPodMemoryUsage critical alert on bcgov-di-plg-loki-0 is
not real memory pressure. On the Silver cluster's cgroup v1 nodes,
container_memory_working_set_bytes includes reclaimable kernel slab
(dentry/inode caches). Loki's constant chunk create/delete churn inflates
that slab until it nears the 2Gi limit, while Loki's actual Go heap is only
~100Mi. The kernel reclaims the slab before any OOM, so the pod sits near
90% working-set indefinitely with zero restarts. Prior memory-limit bumps
could never fix this (slab just refills the larger limit), and cutting
chunk_idle_period to 5m made it worse by increasing file churn.

Two changes:

- prometheus-rule-crd.yaml: exclude the Loki pod from HighPodMemoryUsage
  (working-set is a meaningless signal for it under cgroup v1). A real Loki
  OOM is still caught by the PodInErrorState (OOMKilled) alert. All other
  pods keep the early-warning alert.

- loki-configmap.yaml: raise chunk_idle_period 5m -> 1h, add explicit
  max_chunk_age 2h, and enlarge chunk_target_size to 2Mi so Loki writes
  far fewer chunk files, reducing the dentry/inode slab churn at its source.
  Heap headroom (~100Mi of 2Gi) makes the longer idle period safe.

Documented the cgroup-v1 accounting artifact and the exclusion rationale in
docs-md/ALERTING.md. Structural fix is migrating nodes to cgroup v2.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: alex-struk

deployments/openshift/helm/plg/templates/loki-configmap.yaml

@@ -39,10 +39,16 @@ data:
       ingestion_burst_size_mb: 8
 
     ingester:
-      # Flush idle log streams to disk after 5m instead of 30m to cap the
-      # ingester's in-memory chunk footprint (mitigates Loki high-memory alerts).
-      chunk_idle_period: 5m
-      chunk_target_size: 1536000
+      # Hold idle streams in memory longer and flush larger chunks so Loki
+      # creates far fewer chunk files. Fewer file create/delete cycles means
+      # less kernel dentry/inode slab churn, which is the real driver of the
+      # cgroup-v1 working-set growth behind past high-memory alerts. Loki's
+      # heap has ample headroom (~100Mi used of a 2Gi limit), so a longer
+      # idle period is safe. (An earlier 5m idle period reduced heap the wrong
+      # way and increased file churn.)
+      chunk_idle_period: 1h
+      max_chunk_age: 2h
+      chunk_target_size: 2097152
 
     compactor:
       working_directory: /loki/compactor

deployments/openshift/helm/plg/templates/prometheus-rule-crd.yaml

@@ -24,12 +24,20 @@ spec:
             summary: High CPU usage detected on a pod
             description: A pod in namespace {{ .Release.Namespace }} is using more than 80% of its CPU limit. Check the pod labels in this alert for the affected pod name.
 
+        # The Loki pod is deliberately excluded from this alert. On cgroup v1
+        # nodes, container_memory_working_set_bytes includes reclaimable kernel
+        # slab (dentry/inode caches). Loki's chunk create/delete churn inflates
+        # that slab until it approaches the pod's memory limit, even though
+        # Loki's actual heap stays around ~100Mi and the kernel reclaims the
+        # slab before any OOM. Working-set therefore false-fires at >90% for
+        # Loki with no real memory pressure. A genuine Loki OOM is still caught
+        # by the PodInErrorState alert below (reason=OOMKilled).
         - alert: HighPodMemoryUsage
           expr: >-
             (
-              sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}", container!=""}) by (pod)
+              sum(container_memory_working_set_bytes{namespace="{{ .Release.Namespace }}", container!="", pod!~"{{ include "plg.loki.fullname" . }}-.*"}) by (pod)
               /
-              sum(kube_pod_container_resource_limits{namespace="{{ .Release.Namespace }}", resource="memory"}) by (pod)
+              sum(kube_pod_container_resource_limits{namespace="{{ .Release.Namespace }}", resource="memory", pod!~"{{ include "plg.loki.fullname" . }}-.*"}) by (pod)
             ) > 0.9
           for: 2m
           labels:

docs-md/ALERTING.md

@@ -97,6 +97,22 @@ Static rules that don't use the shared logger counters (e.g. HTTP error rate, sl
 
 ---
 
+## Infrastructure Alert Rules
+
+Infrastructure alerts (pod CPU/memory, PVC capacity, crash/OOM state, Temporal queue depth) are **not** generated from `alert-thresholds.ts`. They are hand-maintained in the Helm template `deployments/openshift/helm/plg/templates/prometheus-rule-crd.yaml`, which renders a `PrometheusRule` CRD carrying the `role: alert-rules` label so OpenShift User Workload Monitoring picks it up. Edit that template directly; the `npm run generate:alert-rules` generator does not touch it.
+
+### Loki is excluded from `HighPodMemoryUsage`
+
+The `HighPodMemoryUsage` rule fires on `container_memory_working_set_bytes / limit > 0.9`. The Loki pod (`<release>-loki-*`) is deliberately excluded via a `pod!~` matcher.
+
+**Why:** the Silver cluster nodes run **cgroup v1**, where `container_memory_working_set_bytes` includes reclaimable kernel slab (dentry/inode caches). Loki continuously creates and deletes chunk files on its filesystem volume (ingester flushes + compaction + retention deletes), which inflates that slab until it approaches the pod's memory limit — even though Loki's actual Go heap stays around ~100 Mi of its 2 Gi limit. The kernel reclaims the slab before any OOM, so working-set sits near 90% indefinitely with no real memory pressure and no restarts. Left in, the rule pages `critical` on a non-problem.
+
+A genuine Loki OOM is still caught by the `PodInErrorState` rule (`reason=OOMKilled`), so no real failure mode is lost by the exclusion. The companion change in `loki-configmap.yaml` (longer `chunk_idle_period`, explicit `max_chunk_age`, larger `chunk_target_size`) reduces the file churn that drives the slab in the first place.
+
+This is a cgroup-v1 accounting artifact; migrating the nodes to cgroup v2 would resolve it structurally and let the exclusion be removed. See Red Hat solution 7126558 for the underlying `xfs_inode`/dentry slab behavior.
+
+---
+
 ## In-App Alerting
 
 Any backend-services or Temporal worker code can raise an alert condition by logging at `warn` or `error` level with an `alertType` in the log context.