feat(synapse): Add support for external Redis configuration

[pkozakCC]

This adds the synapse.externalRedis configuration option, allowing users to connect Synapse workers to managed Redis services (AWS Elasticache, Azure Cache for Redis, etc.) instead of using the internal Redis deployment.

When externalRedis is configured:

• The internal Redis Deployment, Service, ConfigMap, and ServiceAccount are automatically skipped
• Synapse's homeserver.yaml is configured with the external Redis host, port, and optionally password/TLS settings

This follows the same pattern as synapse.postgres for external PostgreSQL.

Configuration options:

• synapse.externalRedis.host (required): Redis host address
• synapse.externalRedis.port (default: 6379): Redis port
• synapse.externalRedis.db (default: 0): Redis database number
• synapse.externalRedis.password: Password (inline value or external secret)
• synapse.externalRedis.tls (default: false): Enable TLS

Fixes: N/A

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[benbz]

Hi, thanks for raising this PR. A few things:

• As asked in [FR] Allow the use of an external Redis Installation #657, what functionality are you hoping to get from an external Redis that you don't get for the one in the chart?
• CLA will need to be signed
• Hookshot would need to gain the settings for an external Redis
  • When using an external Postgres, each Postgres-using-component uses the chart Postgres unless it has been independently configured with an external Postgres (could be the same Postgres, could be a different one)
• Settings should be named synapse.redis to match synapse.postgres

Until the 1st 2 points can be addressed I wouldn't spend effort on the last 2 points.

[pkozakCC]

@benbz

• As asked in [FR] Allow the use of an external Redis Installation #657, what functionality are you hoping to get from an external Redis that you don't get for the one in the chart?

1. High-availability, which cannot be achieved in current implementation
2. The ability to configure redis instances according to my requirements
3. Resource optimization
4. Use elasticache instance, that is shared with other applications

• CLA will need to be signed

It's signed, but for some reason github doesn't see it.

[benbz]

@benbz

• As asked in [FR] Allow the use of an external Redis Installation #657, what functionality are you hoping to get from an external Redis that you don't get for the one in the chart?

1. High-availability, which cannot be achieved in current implementation

My understanding is that high-availability can't be achieved without Synapse changes, i.e. element-hq/synapse#16984. Is that not correct?

2. The ability to configure redis instances according to my requirements
3. Resource optimization

What requirements/optimisations are you thinking of? As the Redis isn't being used as a large cache (primarily pub-sub and a few small caches) there doesn't seem much need to tweak the Redis config. If there are things that are worth tweaking, that feels like a case for improving the in-chart Redis config.

4. Use elasticache instance, that is shared with other applications

The in-chart Redis really doesn't use much resources; no storage, 50Mi memory and 50 milli CPU by default.

I'm not totally opposed to this PR, just I don't see much value in an external Redis at present vs the maintenance burden it will add (compared to an external Postgres where there are clear benefits). I'm happy to be convinced

• CLA will need to be signed

It's signed, but for some reason github doesn't see it.

I can see it on the CLA assistant backend. I'll approve the workflows to run, but I won't be able to merge it without that getting resolved. I'll keep looking on my side, but potentially it will resolve itself next time you push

[github-actions]

dyff of changes in rendered templates of CI manifests

Full contents of manifests and dyffs are available in https://github.com/element-hq/ess-helm/actions/runs/22173004394/artifacts/5569057626

hookshot-secrets-externally-values.yaml

@@ ConfigMap/ess-ci/release-name-hookshot - data.config-override.yaml @@
  bridge:
    domain: "ess.localhost"
    port: 9993
    bindAddress: 0.0.0.0
  
  passFile: /secrets/release-name-hookshot-external-passkey/passfile
  
  cache:
-   redisUri: "redis://release-name-redis.ess-ci.svc.cluster.local.:6379"
+   redisUri: "redis://external-redis.example.com:6379/0"
+   redisPassword: "${HOOKSHOT_REDIS_PASSWORD}"
  
  logging:
    level: info
  
  
  [20 lines unchanged)]
  
  
  
  widgets:
    publicUrl: https://hookshot.ess.localhost/widgetapi/v1/static


@@ ConfigMap/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_configmap.yaml
- apiVersion: v1
- kind: ConfigMap
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- data:
-   redis.conf: |
-     # Do not require a password
-     protected-mode no
-     port 6379
- 
-     tcp-backlog 511
-     tcp-keepalive 300
- 
-     # Never close the connection
-     timeout 0
- 
-     # We run the redis in a container so disable both of these
-     daemonize no
-     supervised no
- 
-     loglevel notice
-     logfile ''
- 
-     databases 16
-     always-show-logo no
-     stop-writes-on-bgsave-error yes
- 
-     # We never save to the disk
-     save ''
- 
-     replica-serve-stale-data yes
-     replica-read-only yes
-     repl-diskless-sync no
-     repl-diskless-sync-delay 5
-     repl-diskless-load disabled
-     repl-disable-tcp-nodelay no
-     replica-priority 100
-     acllog-max-len 128
- 
-     lazyfree-lazy-eviction no
-     lazyfree-lazy-expire no
-     lazyfree-lazy-server-del no
-     replica-lazy-flush no
- 
-     lazyfree-lazy-user-del no
- 
-     lazyfree-lazy-user-flush no
-     oom-score-adj no
-     oom-score-adj-values 0 200 800
- 
-     disable-thp yes
- 
-     appendonly no
-     appendfilename 'appendonly.aof'
-     appendfsync everysec
- 
-     no-appendfsync-on-rewrite no
- 
-     auto-aof-rewrite-percentage 100
-     auto-aof-rewrite-min-size 64mb
-     aof-load-truncated yes
-     aof-use-rdb-preamble yes
-     lua-time-limit 5000
-     slowlog-log-slower-than 10000
-     slowlog-max-len 128
-     latency-monitor-threshold 0
-     notify-keyspace-events ""
-     hash-max-ziplist-entries 512
-     hash-max-ziplist-value 64
-     list-max-ziplist-size -2
-     list-compress-depth 0
-     set-max-intset-entries 512
-     zset-max-ziplist-entries 128
-     zset-max-ziplist-value 64
-     hll-sparse-max-bytes 3000
-     stream-node-max-bytes 4096
-     stream-node-max-entries 100
-     activerehashing yes
-     client-output-buffer-limit normal 0 0 0
-     client-output-buffer-limit replica 256mb 64mb 60
-     client-output-buffer-limit pubsub 32mb 8mb 60
- 
-     # Hz is the freuqency at which background tasks are performed, we keep this low to save CPU
-     hz 1
- 
-     # The hz value is increased to scale with the number of clients connected.
-     dynamic-hz yes
- 
-     aof-rewrite-incremental-fsync yes
-     rdb-save-incremental-fsync yes
-     jemalloc-bg-thread yes
- 
-     maxmemory 40mb
-     maxmemory-policy allkeys-lru


@@ Deployment/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_deployment.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-     k8s.element.io/redis-config-hash: "3034b3cfe78419348e36bb348fad98b46a736334"
-   name: release-name-redis
-   namespace: ess-ci
- spec:
-   replicas: 1
-   selector:
-     matchLabels:
-       app.kubernetes.io/instance: release-name-redis
-   strategy:
-     type: RollingUpdate
-     rollingUpdate:
-       maxSurge: 2
-       maxUnavailable: 0
-   template:
-     metadata:
-       labels:
-         app.kubernetes.io/managed-by: Helm
-         app.kubernetes.io/part-of: matrix-stack
-         app.kubernetes.io/component: matrix-pubsub-small-cache
-         app.kubernetes.io/name: redis
-         app.kubernetes.io/instance: release-name-redis
-         app.kubernetes.io/version: "7.4-alpine"
-         k8s.element.io/redis-config-hash: "3034b3cfe78419348e36bb348fad98b46a736334"
-     spec:
-       automountServiceAccountToken: false
-       serviceAccountName: release-name-redis
-       securityContext:
-         fsGroup: 10002
-         runAsGroup: 10002
-         runAsNonRoot: true
-         runAsUser: 10002
-         seccompProfile:
-           type: RuntimeDefault
-         supplementalGroups: []
-       topologySpreadConstraints:
-         - labelSelector:
-             matchLabels:
-               app.kubernetes.io/instance: release-name-redis
-           matchLabelKeys:
-             - pod-template-hash
-           maxSkew: 1
-           topologyKey: kubernetes.io/hostname
-           whenUnsatisfiable: ScheduleAnyway
-       containers:
-         - name: redis
-           args:
-             - "/config/redis.conf"
-           image: "docker.io/library/redis:7.4-alpine"
-           imagePullPolicy: Always
-           securityContext:
-             allowPrivilegeEscalation: false
-             capabilities:
-               drop:
-                 - ALL
-             readOnlyRootFilesystem: true
-           ports:
-             - containerPort: 6379
-               name: redis
-               protocol: TCP
-           startupProbe:
-             failureThreshold: 5
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             tcpSocket:
-               port: redis
-           livenessProbe:
-             failureThreshold: 3
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             tcpSocket:
-               port: redis
-           readinessProbe:
-             failureThreshold: 3
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             exec:
-               command:
-                 - redis-cli
-                 - ping
-           resources:
-             limits:
-               memory: 50Mi
-             requests:
-               cpu: 50m
-               memory: 50Mi
-           volumeMounts:
-             - mountPath: /config/redis.conf
-               name: config
-               readOnly: true
-               subPath: redis.conf
-       restartPolicy: Always
-       volumes:
-         - configMap:
-             name: "release-name-redis"
-             defaultMode: 420
-           name: config


@@ Service/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_service.yaml
- apiVersion: v1
- kind: Service
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- spec:
-   type: ClusterIP
-   ipFamilyPolicy: PreferDualStack
-   ports:
-     - port: 6379
-       targetPort: redis
-       name: redis
-   selector:
-     app.kubernetes.io/instance: "release-name-redis"


@@ ServiceAccount/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_service_account.yaml
- apiVersion: v1
- kind: ServiceAccount
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- automountServiceAccountToken: false



@@ StatefulSet/ess-ci/release-name-hookshot - spec.template.spec.initContainers.render-config @@
+   env:
+   - name: HOOKSHOT_REDIS_PASSWORD
+     value: "{{ readfile \"/secrets/release-name-redis-credentials/password\" | quote }}"

@@ StatefulSet/ess-ci/release-name-hookshot - spec.template.spec.initContainers.render-config.volumeMounts @@
+   - name: secret-b01d9752db53
+     mountPath: /secrets/release-name-redis-credentials
+     readOnly: true

@@ StatefulSet/ess-ci/release-name-hookshot - spec.template.spec.containers.hookshot.volumeMounts @@
+   - name: secret-b01d9752db53
+     mountPath: /secrets/release-name-redis-credentials
+     readOnly: true

@@ StatefulSet/ess-ci/release-name-hookshot - spec.template.spec.volumes @@
+   - name: secret-b01d9752db53
+     secret:
+       secretName: release-name-redis-credentials

hookshot-secrets-in-helm-values.yaml

@@ ConfigMap/ess-ci/release-name-hookshot - data.config-override.yaml @@
  bridge:
    domain: "ess.localhost"
    port: 9993
    bindAddress: 0.0.0.0
  
  passFile: /secrets/release-name-hookshot/RSA_PASSKEY
  
  cache:
-   redisUri: "redis://release-name-redis.ess-ci.svc.cluster.local.:6379"
+   redisUri: "redis://external-redis.example.com:6379/0"
+   redisPassword: "${HOOKSHOT_REDIS_PASSWORD}"
  
  logging:
    level: info
  
  
  [20 lines unchanged)]
  
  
  
  widgets:
    publicUrl: https://hookshot.ess.localhost/widgetapi/v1/static


@@ ConfigMap/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_configmap.yaml
- apiVersion: v1
- kind: ConfigMap
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- data:
-   redis.conf: |
-     # Do not require a password
-     protected-mode no
-     port 6379
- 
-     tcp-backlog 511
-     tcp-keepalive 300
- 
-     # Never close the connection
-     timeout 0
- 
-     # We run the redis in a container so disable both of these
-     daemonize no
-     supervised no
- 
-     loglevel notice
-     logfile ''
- 
-     databases 16
-     always-show-logo no
-     stop-writes-on-bgsave-error yes
- 
-     # We never save to the disk
-     save ''
- 
-     replica-serve-stale-data yes
-     replica-read-only yes
-     repl-diskless-sync no
-     repl-diskless-sync-delay 5
-     repl-diskless-load disabled
-     repl-disable-tcp-nodelay no
-     replica-priority 100
-     acllog-max-len 128
- 
-     lazyfree-lazy-eviction no
-     lazyfree-lazy-expire no
-     lazyfree-lazy-server-del no
-     replica-lazy-flush no
- 
-     lazyfree-lazy-user-del no
- 
-     lazyfree-lazy-user-flush no
-     oom-score-adj no
-     oom-score-adj-values 0 200 800
- 
-     disable-thp yes
- 
-     appendonly no
-     appendfilename 'appendonly.aof'
-     appendfsync everysec
- 
-     no-appendfsync-on-rewrite no
- 
-     auto-aof-rewrite-percentage 100
-     auto-aof-rewrite-min-size 64mb
-     aof-load-truncated yes
-     aof-use-rdb-preamble yes
-     lua-time-limit 5000
-     slowlog-log-slower-than 10000
-     slowlog-max-len 128
-     latency-monitor-threshold 0
-     notify-keyspace-events ""
-     hash-max-ziplist-entries 512
-     hash-max-ziplist-value 64
-     list-max-ziplist-size -2
-     list-compress-depth 0
-     set-max-intset-entries 512
-     zset-max-ziplist-entries 128
-     zset-max-ziplist-value 64
-     hll-sparse-max-bytes 3000
-     stream-node-max-bytes 4096
-     stream-node-max-entries 100
-     activerehashing yes
-     client-output-buffer-limit normal 0 0 0
-     client-output-buffer-limit replica 256mb 64mb 60
-     client-output-buffer-limit pubsub 32mb 8mb 60
- 
-     # Hz is the freuqency at which background tasks are performed, we keep this low to save CPU
-     hz 1
- 
-     # The hz value is increased to scale with the number of clients connected.
-     dynamic-hz yes
- 
-     aof-rewrite-incremental-fsync yes
-     rdb-save-incremental-fsync yes
-     jemalloc-bg-thread yes
- 
-     maxmemory 40mb
-     maxmemory-policy allkeys-lru


@@ Deployment/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_deployment.yaml
- apiVersion: apps/v1
- kind: Deployment
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-     k8s.element.io/redis-config-hash: "3034b3cfe78419348e36bb348fad98b46a736334"
-   name: release-name-redis
-   namespace: ess-ci
- spec:
-   replicas: 1
-   selector:
-     matchLabels:
-       app.kubernetes.io/instance: release-name-redis
-   strategy:
-     type: RollingUpdate
-     rollingUpdate:
-       maxSurge: 2
-       maxUnavailable: 0
-   template:
-     metadata:
-       labels:
-         app.kubernetes.io/managed-by: Helm
-         app.kubernetes.io/part-of: matrix-stack
-         app.kubernetes.io/component: matrix-pubsub-small-cache
-         app.kubernetes.io/name: redis
-         app.kubernetes.io/instance: release-name-redis
-         app.kubernetes.io/version: "7.4-alpine"
-         k8s.element.io/redis-config-hash: "3034b3cfe78419348e36bb348fad98b46a736334"
-     spec:
-       automountServiceAccountToken: false
-       serviceAccountName: release-name-redis
-       securityContext:
-         fsGroup: 10002
-         runAsGroup: 10002
-         runAsNonRoot: true
-         runAsUser: 10002
-         seccompProfile:
-           type: RuntimeDefault
-         supplementalGroups: []
-       topologySpreadConstraints:
-         - labelSelector:
-             matchLabels:
-               app.kubernetes.io/instance: release-name-redis
-           matchLabelKeys:
-             - pod-template-hash
-           maxSkew: 1
-           topologyKey: kubernetes.io/hostname
-           whenUnsatisfiable: ScheduleAnyway
-       containers:
-         - name: redis
-           args:
-             - "/config/redis.conf"
-           image: "docker.io/library/redis:7.4-alpine"
-           imagePullPolicy: Always
-           securityContext:
-             allowPrivilegeEscalation: false
-             capabilities:
-               drop:
-                 - ALL
-             readOnlyRootFilesystem: true
-           ports:
-             - containerPort: 6379
-               name: redis
-               protocol: TCP
-           startupProbe:
-             failureThreshold: 5
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             tcpSocket:
-               port: redis
-           livenessProbe:
-             failureThreshold: 3
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             tcpSocket:
-               port: redis
-           readinessProbe:
-             failureThreshold: 3
-             periodSeconds: 10
-             successThreshold: 1
-             timeoutSeconds: 1
-             exec:
-               command:
-                 - redis-cli
-                 - ping
-           resources:
-             limits:
-               memory: 50Mi
-             requests:
-               cpu: 50m
-               memory: 50Mi
-           volumeMounts:
-             - mountPath: /config/redis.conf
-               name: config
-               readOnly: true
-               subPath: redis.conf
-       restartPolicy: Always
-       volumes:
-         - configMap:
-             name: "release-name-redis"
-             defaultMode: 420
-           name: config



@@ Secret/ess-ci/release-name-hookshot - data @@
+   REDIS_PASSWORD: dGVzdC1yZWRpcy1wYXNzd29yZA==


@@ Service/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_service.yaml
- apiVersion: v1
- kind: Service
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- spec:
-   type: ClusterIP
-   ipFamilyPolicy: PreferDualStack
-   ports:
-     - port: 6379
-       targetPort: redis
-       name: redis
-   selector:
-     app.kubernetes.io/instance: "release-name-redis"


@@ ServiceAccount/ess-ci/release-name-redis @@
- ---
- # Source: matrix-stack/templates/redis/redis_service_account.yaml
- apiVersion: v1
- kind: ServiceAccount
- metadata:
-   labels:
-     helm.sh/chart: "matrix-stack-26.2.3-dev"
-     app.kubernetes.io/managed-by: Helm
-     app.kubernetes.io/part-of: matrix-stack
-     app.kubernetes.io/component: matrix-pubsub-small-cache
-     app.kubernetes.io/name: redis
-     app.kubernetes.io/instance: release-name-redis
-     app.kubernetes.io/version: "7.4-alpine"
-   name: release-name-redis
-   namespace: ess-ci
- automountServiceAccountToken: false



@@ StatefulSet/ess-ci/release-name-hookshot - spec.template.spec.initContainers.render-config @@
+   env:
+   - name: HOOKSHOT_REDIS_PASSWORD
+     value: "{{ readfile \"/secrets/release-name-hookshot/REDIS_PASSWORD\" | quote }}"

synapse-redis-secrets-externally-values.yaml (added)

ConfigMap/ess-ci/release-name-haproxy added
ConfigMap/ess-ci/release-name-postgres added
ConfigMap/ess-ci/release-name-synapse-haproxy added
ConfigMap/ess-ci/release-name-synapse-hook added
ConfigMap/ess-ci/release-name-synapse added
Deployment/ess-ci/release-name-haproxy added
Ingress/ess-ci/release-name-synapse added
Job/ess-ci/release-name-init-secrets added
Job/ess-ci/release-name-synapse-check-config added
PersistentVolumeClaim/ess-ci/release-name-postgres-data added
PersistentVolumeClaim/ess-ci/release-name-synapse-media added
Role/ess-ci/release-name-init-secrets added
RoleBinding/ess-ci/release-name-init-secrets added
Secret/ess-ci/release-name-synapse-hook added
Secret/ess-ci/release-name-synapse added
Service/ess-ci/release-name-haproxy added
Service/ess-ci/release-name-postgres added
Service/ess-ci/release-name-synapse-appservice added
Service/ess-ci/release-name-synapse-client-reader added
Service/ess-ci/release-name-synapse-event-persist added
Service/ess-ci/release-name-synapse-fed-reader added
Service/ess-ci/release-name-synapse-main added
Service/ess-ci/release-name-synapse added
ServiceAccount/ess-ci/release-name-haproxy added
ServiceAccount/ess-ci/release-name-init-secrets added
ServiceAccount/ess-ci/release-name-postgres added
ServiceAccount/ess-ci/release-name-synapse-check-config added
ServiceAccount/ess-ci/release-name-synapse added
ServiceMonitor/ess-ci/release-name-haproxy added
ServiceMonitor/ess-ci/release-name-postgres added
ServiceMonitor/ess-ci/release-name-synapse added
StatefulSet/ess-ci/release-name-postgres added
StatefulSet/ess-ci/release-name-synapse-appservice added
StatefulSet/ess-ci/release-name-synapse-client-reader added
StatefulSet/ess-ci/release-name-synapse-event-persist added
StatefulSet/ess-ci/release-name-synapse-fed-reader added
StatefulSet/ess-ci/release-name-synapse-main added

synapse-redis-secrets-in-helm-values.yaml (added)

ConfigMap/ess-ci/release-name-haproxy added
ConfigMap/ess-ci/release-name-postgres added
ConfigMap/ess-ci/release-name-synapse-haproxy added
ConfigMap/ess-ci/release-name-synapse-hook added
ConfigMap/ess-ci/release-name-synapse added
Deployment/ess-ci/release-name-haproxy added
Ingress/ess-ci/release-name-synapse added
Job/ess-ci/release-name-init-secrets added
Job/ess-ci/release-name-synapse-check-config added
PersistentVolumeClaim/ess-ci/release-name-postgres-data added
PersistentVolumeClaim/ess-ci/release-name-synapse-media added
Role/ess-ci/release-name-init-secrets added
RoleBinding/ess-ci/release-name-init-secrets added
Secret/ess-ci/release-name-synapse-hook added
Secret/ess-ci/release-name-synapse added
Service/ess-ci/release-name-haproxy added
Service/ess-ci/release-name-postgres added
Service/ess-ci/release-name-synapse-appservice added
Service/ess-ci/release-name-synapse-client-reader added
Service/ess-ci/release-name-synapse-event-persist added
Service/ess-ci/release-name-synapse-fed-reader added
Service/ess-ci/release-name-synapse-main added
Service/ess-ci/release-name-synapse added
ServiceAccount/ess-ci/release-name-haproxy added
ServiceAccount/ess-ci/release-name-init-secrets added
ServiceAccount/ess-ci/release-name-postgres added
ServiceAccount/ess-ci/release-name-synapse-check-config added
ServiceAccount/ess-ci/release-name-synapse added
ServiceMonitor/ess-ci/release-name-haproxy added
ServiceMonitor/ess-ci/release-name-postgres added
ServiceMonitor/ess-ci/release-name-synapse added
StatefulSet/ess-ci/release-name-postgres added
StatefulSet/ess-ci/release-name-synapse-appservice added
StatefulSet/ess-ci/release-name-synapse-client-reader added
StatefulSet/ess-ci/release-name-synapse-event-persist added
StatefulSet/ess-ci/release-name-synapse-fed-reader added
StatefulSet/ess-ci/release-name-synapse-main added

[pkozakCC]

@benbz

My understanding is that high-availability can't be achieved without Synapse changes, i.e. element-hq/synapse#16984. Is that not correct?

This is the primary motivation for this PR. Supporting an external Redis instance allows users to leverage managed services that handle high availability (HA) natively. This includes solutions such as AWS ElastiCache, Azure Cache for Redis, or Google Cloud Memorystore. No changes to the Synapse core are required, as the application simply connects to the provided endpoint, which serves as a proxy/gateway to the managed, highly available cluster.

[Bo-Ba]

The in-chart Redis really doesn't use much resources; no storage, 50Mi memory and 50 milli CPU by default.
... I don't see much value in an external Redis at present vs the maintenance burden it will add ... I'm happy to be convinced
... Redis isn't being used as a large cache (primarily pub-sub and a few small caches) ...

Precisely because Redis is used here in a fairly standard way (primarily pub-sub and a few small caches), I don’t see a strong reason to effectively require a dedicated Redis deployment per matrix chart when an operator already has a Redis service available. In many setups there’s already a Redis instance in place (often used as a shared cache elsewhere) with established monitoring/alerting, backup/restore practices where applicable, and known operational behaviour under load.
Reusing that reduces the number of moving parts and gives more options when things go wrong.

And as @pkozakCC mentioned, this also allows using a managed Redis service that provides availability/failover behind a stable endpoint (without implying Sentinel/Redis Cluster support in Synapse).

[benbz]

@benbz

My understanding is that high-availability can't be achieved without Synapse changes, i.e. element-hq/synapse#16984. Is that not correct?

This is the primary motivation for this PR. Supporting an external Redis instance allows users to leverage managed services that handle high availability (HA) natively. This includes solutions such as AWS ElastiCache, Azure Cache for Redis, or Google Cloud Memorystore. No changes to the Synapse core are required, as the application simply connects to the provided endpoint, which serves as a proxy/gateway to the managed, highly available cluster.

That feels like a reasonable reason to support an external Redis. Happy to progress this PR, but points 3 & 4 from #1029 (comment) will need to be addressed before merging

[benbz]

The 3x integration test failures are either flakes or unrelated to this PR that we need to fix. The other 4 failures are relevant to this PR

[gaelgatelement]

@pkozakCC Using the script scripts/assemble_ci_values_files_from_fragments.sh will update the CI files and remove trailing spaces.