prometheus: enable readOnlyRootFilesystem

The Prometheus container previously required readOnlyRootFilesystem: false
because prom-wrapper writes the Alertmanager configuration file to
/sg_config_prometheus/alertmanager.yml at runtime whenever site config
observability.alerts or SMTP settings change. Additionally, Alertmanager
stores state (silences, notification log) at /alertmanager/.

This change enables readOnlyRootFilesystem: true by:

1. Adding an emptyDir volume mounted at /alertmanager for Alertmanager
   state storage (--storage.path).

2. Setting ALERTMANAGER_CONFIG_PATH=/alertmanager/alertmanager.yml so
   prom-wrapper writes the Alertmanager config to the writable emptyDir
   instead of the read-only image layer at /sg_config_prometheus/.

The /prometheus TSDB data path was already handled by the existing PVC,
and /sg_prometheus_add_ons is already a ConfigMap mount. The baked-in
alert rules at /sg_config_prometheus/ are read-only at runtime (only
written at image build time), so they remain accessible on the
read-only root filesystem.

Resolves: https://github.com/sourcegraph/sourcegraph/issues/34012
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbe72-f13d-7233-aa22-cbeb04d03182

Author: marcleblanc2

charts/sourcegraph/templates/prometheus/prometheus.Deployment.yaml

@@ -47,12 +47,24 @@ spec:
       - name: prometheus
         image: {{ include "sourcegraph.image" (list . "prometheus") }}
         imagePullPolicy: {{ .Values.sourcegraph.image.pullPolicy }}
+        # Seed the Alertmanager config from the baked-in default on first start
+        # so it works with readOnlyRootFilesystem. prom-wrapper then rewrites this
+        # file in place whenever site config changes (see ALERTMANAGER_CONFIG_PATH).
+        command:
+        - /bin/sh
+        - -c
+        - |
+          cp -n /sg_config_prometheus/alertmanager.yml /alertmanager/alertmanager.yml
+          exec /usr/bin/prom-wrapper "$@"
+        - prom-wrapper
         {{- with .Values.prometheus.args }}
         args:
           {{- toYaml . | nindent 8 }}
         {{- end }}
         terminationMessagePolicy: FallbackToLogsOnError
         env:
+        - name: ALERTMANAGER_CONFIG_PATH
+          value: /alertmanager/alertmanager.yml
         {{- range $name, $item := .Values.prometheus.env}}
         - name: {{ $name }}
           {{- $item | toYaml | nindent 10 }}
@@ -72,6 +84,9 @@ spec:
           name: data
         - mountPath: /sg_prometheus_add_ons
           name: config
+        - mountPath: /alertmanager
+          name: data
+          subPath: alertmanager
         {{- if .Values.prometheus.extraVolumeMounts }}
         {{- toYaml .Values.prometheus.extraVolumeMounts | nindent 8 }}
         {{- end }}

charts/sourcegraph/values.yaml

@@ -906,8 +906,6 @@ prometheus:
     allowPrivilegeEscalation: false
     runAsUser: 100
     runAsGroup: 100
-    # Read-only filesystem not supported for the prometheus container,
-    # see [sourcegraph/issues/34012](https://github.com/sourcegraph/sourcegraph/issues/34012) for more information
     readOnlyRootFilesystem: false
   # -- Name used by resources. Does not affect service names or PVCs.
   name: "prometheus"