Nextcloud seems to not be able to connect to redis with default settings

[jvalskis]

Describe your Issue

I have an issue with Nextcloud failing to connect to redis when it's enabled. Here's an excerpt from the logs:

{
  "reqId":"b8E9pguPPMd2bgHNOOj0",
  "level":3,
  "time":"2023-09-26T06:14:19+00:00",
  "remoteAddr":"10.42.5.1",
  "user":"--",
  "app":"PHP",
  "method":"GET",
  "url":"/status.php",
  "message":"session_start(): Failed to read session data: redis (path: tcp://nextcloud-redis-master:6379?auth=\<redacted\>) at /var/www/html/lib/private/Session/Internal.php#222",
  "userAgent":"kube-probe/1.27",
  "version":"27.1.1.0",
  "data":{"app":"PHP"}
}

Which is weird, because when I do nslookup, it seems to resolve fine:

$ nslookup nextcloud-redis-master
Server:         10.43.0.10
Address:        10.43.0.10#53

Name:   nextcloud-redis-master.cloud.svc.cluster.local
Address: 10.43.32.88

What I did as a workaround, was to hardcode the "host" part in redis.config.php as such: 'host' => "nextcloud-redis-master.cloud.svc.cluster.local",. Any ideas what might be wrong here?

Logs and Errors

Describe your Environment

• Kubernetes distribution: k3s

• Helm Version (or App that manages helm): 3.11.0

• Helm Chart Version: 3.5.22

• values.yaml:

nextcloud:
  defaultConfigs:
    redis.config.php: false
  configs:
    redis.config.php: |-
      <?php
      $CONFIG = array (
        'memcache.distributed' => '\OC\Memcache\Redis',
        'memcache.locking' => '\OC\Memcache\Redis',
        'redis' => array(
          'host' => "nextcloud-redis-master.cloud.svc.cluster.local",
          'port' => getenv('REDIS_HOST_PORT') ?: 6379,
          'password' => getenv('REDIS_HOST_PASSWORD'),
        ),
      );
redis:
    enabled: true
    usePassword: true
    auth:
      existingSecret: nextcloud-auth-config
      existingSecretPasswordKey: redis-password
    master:
      persistence:
        storageClass: longhorn-fast-retain
    replica:
      persistence:
        storageClass: longhorn-fast-retain

[jessebot]

I haven't had time to look into this fully, but wanted to give some more details to help others troubleshoot:

Here's where the full name is defined in our templates, specifically _helpers.tpl:

helm/charts/nextcloud/templates/_helpers.tpl

Lines 26 to 32 in 0c6822f

	{{/*
	Create a default fully qualified redis app name.
	We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
	*/}}
	{{- define "nextcloud.redis.fullname" -}}
	{{- printf "%s-%s" .Release.Name "redis" | trunc 63 | trimSuffix "-" -}}
	{{- end -}}

and it gets referenced again here in our _helpers.tpl:

helm/charts/nextcloud/templates/_helpers.tpl

Lines 215 to 232 in 0c6822f

	{{- if .Values.redis.enabled }}
	- name: REDIS_HOST
	value: {{ template "nextcloud.redis.fullname" . }}-master
	- name: REDIS_HOST_PORT
	value: {{ .Values.redis.master.service.ports.redis | quote }}
	{{- if .Values.redis.auth.enabled }}
	{{- if and .Values.redis.auth.existingSecret .Values.redis.auth.existingSecretPasswordKey }}
	- name: REDIS_HOST_PASSWORD
	valueFrom:
	secretKeyRef:
	name: {{ .Values.redis.auth.existingSecret }}
	key: {{ .Values.redis.auth.existingSecretPasswordKey }}
	{{- else }}
	- name: REDIS_HOST_PASSWORD
	value: {{ .Values.redis.auth.password }}
	{{- end }}
	{{- end }}
	{{- end }}

@jvalskis when it is unable to connect, could you please log into the container and check the environment variable called REDIS_HOST? Should be something like this to get it:

# where NEXTCLOUD_POD is the actual name of your nextcloud pod
kubectl exec $NEXTCLOUD_POD -- env | grep REDIS_HOST

[Kaurin]

I can replicate the error on my k3s cluster, but the timestamp shows that it happens within 60 seconds of the pod being started so it could be down to just redis not being ready?

I logged in/logged out a few times and the error does not repeat.

OP error:

{
  "reqId": "6dFzKM6uQLlOijIHHBD0",
  "level": 3,
  "time": "2024-05-07T04:29:34+00:00",
  "remoteAddr": "10.42.0.1",
  "user": "--",
  "app": "PHP",
  "method": "GET",
  "url": "/status.php",
  "message": "session_start(): Failed to read session data: redis (path: tcp://nextcloud-troubleshoot-redis-master:6379?auth=changeme) at /var/www/html/lib/private/Session/Internal.php#213",
  "userAgent": "kube-probe/1.28",
  "version": "29.0.0.19",
  "data": {
    "app": "PHP"
  }
}

Another error I noticed:

{
  "reqId": "sGLjn5exo0J3ZAYR8Fde",
  "level": 3,
  "time": "2024-05-07T04:29:39+00:00",
  "remoteAddr": "10.42.0.1",
  "user": "--",
  "app": "PHP",
  "method": "GET",
  "url": "/status.php",
  "message": "session_start(): Redis connection not available at /var/www/html/lib/private/Session/Internal.php#213",
  "userAgent": "kube-probe/1.28",
  "version": "29.0.0.19",
  "data": {
    "app": "PHP"
  }
}

Again, within 60 seconds of the pods starting. Does not repeat even after multiple login/logout.

values.yml used:

redis:
    enabled: true

EDIT: I can replicate the issue when using secret-based auth. WIP

[Kaurin]

Welp. I don't know if the issue is the same as OPs, but initially my redis password contained quite a few special characters.

Before doing anything else, I changed that password to non-special chars. Uninstall, clean PVCs and Volumes, install again without changing values.yml, and it works fine.

Current, working values.yml

redis:
    enabled: true
    usePassword: true
    auth:
      existingSecret: nextcloud-redis
      existingSecretPasswordKey: redisPw

Note that I still get OP's error message before redis is ready

[ensignavenger]

Getting the same error but using Docker Compose instead of Kubernetes.

My log:

{"reqId":"CsmaTyEdjJhymt1vrdQM",
"level":3,
"time":"2024-08-16T16:19:16+00:00",
"remoteAddr":"myipaddress",
"user":"--",
"app":"PHP",
"method":"POST",
"url":"/",
"message":"session_start(): Failed to read session data: redis (path: tcp://valkey:6379?auth=cIQCrandomcharspassword) at /var/www/html/lib/private/Session/Internal.php#214",
"userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0",
"version":"29.0.4.1",
"data":{"app":"PHP"}}

My compose file:

services:
  postgres:
    image: postgres:16.4
    networks:
      - nextcloud
    expose:
      - "5433"
    volumes:
      - ${DOCKER_VOLUMES}/cloud/postgres/data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
    secrets:
      - postgres_password
      - postgres_user
      - postgres_db

  nextcloud:
    image: nextcloud:29.0.4
    expose:
      - 80
    networks:
      - proxy_web
      - nextcloud
    volumes:
      - ${DOCKER_VOLUMES}/cloud/nextcloud/html:/var/www/html
    environment:
      - POSTGRES_HOST=postgres
      - POSTGRES_DB_FILE=/run/secrets/postgres_db
      - POSTGRES_USER_FILE=/run/secrets/postgres_user
      - POSTGRES_PASSWORD_FILE=/run/secrets/postgres_password
      - REDIS_HOST=valkey
      - REDIS_HOST_PORT=6379
      - REDIS_HOST_PASSWORD_FILE=/run/secrets/valkey_password
    depends_on:
      - postgres
      - valkey
    secrets:
      - postgres_password
      - postgres_user
      - postgres_db
      - valkey_password
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy_web"
      - "traefik.http.routers.nextcloud.rule=Host(`mydomainname`)"
      - "traefik.http.routers.nextcloud.entrypoints=websecure"
      - "traefik.http.routers.nextcloud.tls.certresolver=mytlschallenge"
      - "traefik.http.services.nextcloud.loadBalancer.server.port=80"

  valkey: 
    # image: valkey/valkey:8.0
    image: redis
    expose: 
      - 6379
    networks:
      - nextcloud
    volumes:
      - ${DOCKER_VOLUMES}/cloud/valkey/data:/data

  utils:
    image: nicolaka/netshoot
    command: tail -f /dev/null
    networks:
      - nextcloud
      - proxy_web

secrets:
  nextcloud_admin_password:
    file: .secrets/nextcloud_admin_password.txt
  postgres_password:
    file: .secrets/postgres_password.txt
  postgres_db:
    file: .secrets/postgres_db.txt
  postgres_user:
    file: .secrets/postgres_user.txt 
  valkey_password:
    file: .secrets/valkey_password.txt 

networks:
  proxy_web:
    external: true
  nextcloud:

I use the netshoot container to verify the docker netowrk is working and I can access the redis container with the "valkey" hostname.

Before accessing the site, I give it plenty of time (several minutes) to ensuire everything is started. I wipe the volumes after making any changes (like from valkey to redis) to setup everything fresh.

I verified my password does not contain any special characters (lowercase, uppercase letters and numbers only, no symbols).

Note: I initially set it up to use valkey, an open source fork of redis that should be pretty "drop in" at this point, because redis is now closed source software. However, after having issues, I swapped it out for the official redis image for troubleshooting.

[ensignavenger]

env | grep REDIS_HOST

I followed this suggestion on my setup, and confirmed the envars are set.

REDIS_HOST_PORT=6379
REDIS_HOST_PASSWORD_FILE=/run/secrets/valkey_password
REDIS_HOST=valkey

[jessebot]

@ensignavenger I edited your comments just a bit for syntax highlighting and formatting, so I could read them better. Since you're using docker compose, could you please ask about this in the https://github.com/nextcloud/docker repo? This repo is for the helm chart only. Please feel free to cross link your issue here, so others can benefit from your legwork though.

I'm also not sure that valkey is 100% supported yet, but I am all here for it 🙏

[shadow1runner]

Welp. I don't know if the issue is the same as OPs, but initially my redis password contained quite a few special characters.

Before doing anything else, I changed that password to non-special chars. Uninstall, clean PVCs and Volumes, install again without changing values.yml, and it works fine.

Current, working values.yml

redis:
enabled: true
usePassword: true
auth:
existingSecret: nextcloud-redis
existingSecretPasswordKey: redisPw

Note that I still get OP's error message before redis is ready

Thanks for this!
I was using "special" characters (! and &), too; the latter was url-encoded correctly, but still: removing these and starting over from scratch helped!