Loki + Promtail Memory Bump and other infrastructure tweaks

[dbarkowsky]

Summary

I've manually applied a lot of these changes to the TEST environment. Things seem to be stable. The only question is around whether Loki will continue to max out memory limits.

Changes

• Promtail sidecar memory increased in both backend-services and temporal-worker deployments (request 32Mi → 64Mi, limit 64Mi → 128Mi) to prevent OOMKills caused by Loki backpressure filling the in-memory send queue.
• Logrotate rotation threshold lowered from 50M to 10M in both deployments to keep log files small and reduce peak Promtail buffer demand.
• Loki memory limit raised from 1Gi to 2Gi in values-openshift.yaml, and ingestion rate limits (ingestion_rate_mb: 4, ingestion_burst_size_mb: 8) and ingester flush tuning added to the Loki config to apply back-pressure before memory is exhausted.
• Prometheus pod discovery fixed: added a dedicated ServiceAccount, Role (pods: get/list/watch), and RoleBinding for Prometheus so Kubernetes SD configs no longer fail with a pods is forbidden error against the default service account.
• Loki metrics now scraped by Prometheus: added a loki scrape job using a static target, enabling process_resident_memory_bytes and related queries to work.
• Alertmanager crash on startup fixed: receiver definitions for ches-notifications and teams-notifications are now gated on notificationsEnabled, preventing an empty webhook URL from failing config validation when notifications are disabled.
• ches-adapter resources removed when notifications are disabled: all 5 ches-adapter Helm templates (Deployment, Service, PodDisruptionBudget, logrotate and promtail ConfigMaps) are now gated on notificationsEnabled, fixing InvalidImageName errors caused by rendering the deployment with an empty image repository.
• Grafana deployment strategy set to Recreate to prevent Multi-Attach errors on the RWO PVC when a rolling update starts a new pod before the old one terminates.

Testing

Can check most of this until there is another PR merged that might overwrite it... then will have to check after this is merged.

• No PodInErrorState or HighPodMemoryUsage alerts are firing in the test namespace.
• Alertmanager and CHES adapter are running without alerts.

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

• a self-review of my code
• commented code particularly in hard-to-understand areas
• corresponding changes to the documentation where required
• changes tested to the best of my ability
• no new errors or non-functional code