prometheus: enable readOnlyRootFilesystem

The Prometheus container previously required readOnlyRootFilesystem: false
because prom-wrapper writes the Alertmanager configuration file to
/sg_config_prometheus/alertmanager.yml at runtime whenever site config
observability.alerts or SMTP settings change. Additionally, Alertmanager
stores state (silences, notification log) at /alertmanager/.

This change enables readOnlyRootFilesystem: true by:

1. Adding an emptyDir volume mounted at /alertmanager for Alertmanager
   state storage (--storage.path).

2. Setting ALERTMANAGER_CONFIG_PATH=/alertmanager/alertmanager.yml so
   prom-wrapper writes the Alertmanager config to the writable emptyDir
   instead of the read-only image layer at /sg_config_prometheus/.

The /prometheus TSDB data path was already handled by the existing PVC,
and /sg_prometheus_add_ons is already a ConfigMap mount. The baked-in
alert rules at /sg_config_prometheus/ are read-only at runtime (only
written at image build time), so they remain accessible on the
read-only root filesystem.

Resolves: https://github.com/sourcegraph/sourcegraph/issues/34012
Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-019dbe72-f13d-7233-aa22-cbeb04d03182

Author: marcleblanc2

charts/sourcegraph/templates/prometheus/prometheus.Deployment.yaml

@@ -53,6 +53,8 @@ spec:
         {{- end }}
         terminationMessagePolicy: FallbackToLogsOnError
         env:
+        - name: ALERTMANAGER_CONFIG_PATH
+          value: /alertmanager/alertmanager.yml
         {{- range $name, $item := .Values.prometheus.env}}
         - name: {{ $name }}
           {{- $item | toYaml | nindent 10 }}
@@ -72,6 +74,8 @@ spec:
           name: data
         - mountPath: /sg_prometheus_add_ons
           name: config
+        - mountPath: /alertmanager
+          name: alertmanager-data
         {{- if .Values.prometheus.extraVolumeMounts }}
         {{- toYaml .Values.prometheus.extraVolumeMounts | nindent 8 }}
         {{- end }}
@@ -104,6 +108,8 @@ spec:
         configMap:
           defaultMode: 0777
           name: {{ default .Values.prometheus.name .Values.prometheus.existingConfig }}
+      - name: alertmanager-data
+        emptyDir: {}
       {{- if .Values.prometheus.extraVolumes }}
       {{- toYaml .Values.prometheus.extraVolumes | nindent 6 }}
       {{- end }}

charts/sourcegraph/values.yaml

@@ -906,8 +906,6 @@ prometheus:
     allowPrivilegeEscalation: false
     runAsUser: 100
     runAsGroup: 100
-    # Read-only filesystem not supported for the prometheus container,
-    # see [sourcegraph/issues/34012](https://github.com/sourcegraph/sourcegraph/issues/34012) for more information
     readOnlyRootFilesystem: false
   # -- Name used by resources. Does not affect service names or PVCs.
   name: "prometheus"