Stuck at "Initializing Nextcloud..." when attached to NFS PVC

[somerandow]

Doing my best to dupe helm/charts#22920 over to the new repo as I am experiencing this issue as well. I have refined the details a bit, as this issue appears to be specifically related to NFS-based storage.

Describe the bug

When bringing up the nextcloud pod via the helm chart, the logs show the pod as being stuck at:

2020-08-31T19:00:42.054297154Z Configuring Redis as session handler
2020-08-31T19:00:42.098305129Z Initializing nextcloud 19.0.1.1 ...

Even backing out the liveness/readiness probes to over 5 minutes does not give
If I instead switch the PVC to my storageClass for Rancher Longhorn (iSCSI) for example, the nextcloud install initializes in seconds.

Version of Helm and Kubernetes:

helm: v3.3.0
kubernetes: v1.18.6

Which chart:

nextcloud/helm

What happened:

• Namespace is created.
• Helm creates NFS PVC, or it is created manually
• Helm instantiates Nextcloud pod
• Nextcloud pod attaches PVC, and starts
• Nextcloud container is stuck at the above line

What you expected to happen:

Nextcloud finishes initialization
Nextcloud files appear with correct permissions on NFS volume

How to reproduce it (as minimally and precisely as possible):

Set up an NFS provisioner:

helm install stable/nfs-client-provisioner nfs  \
--set nfs.server=x.x.x.x --set nfs.path=<path>

OR
Configure an NFS PV and PVC manually

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nextcloud-data
  labels:
    app: cloud
    type: data
spec:
  capacity:
    storage: 100Ti
  nfs:
    path: <path>
    server: <server>
  mountOptions:
    - async
    - nfsvers=4.2
    - noatime
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-manual
  volumeMode: Filesystem
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nextcloud-data
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Ti
  storageClassName: nfs-manual
  volumeMode: Filesystem
  selector:
    matchLabels:
      app: cloud
      type: data

Install nextcloud
helm install -f values.yaml nextcloud/helm nextcloud --namespace=nextcloud

values.yaml:

image:
  repository: nextcloud
  tag: 19
readinessProbe:
  initialDelaySeconds: 560
livenessProbe:
  initialDelaySeconds: 560
resources:
  requests:
    cpu: 200m
    memory: 500Mi
  limits:
    cpu: 2
    memory: 1Gi
ingress:
  enabled: true
  annotations:
    cert-manager.io/cluster-issuer: acme
    kubernetes.io/ingress.class: nginx
    # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  hosts:
    - "cloud.myhost.com"
  tls:
    - hosts:
        - "cloud.myhost.com"
      secretName: prod-cert
  path: /
nextcloud:
  username: admin
  password: admin1
  # datadir: /mnt/data
  host: "cloud.myhost.com"
internalDatabase:
  enabled: true
externalDatabase:
  enabled: false
persistence:
  enabled: true
  # accessMode: ReadWriteMany
  # storageClass: nfs-client if creating via provisioner
  existingClaim: nextcloud-data # comment out if creating new PVC via provisioner

[somerandow]

I will add as well that my example PV above includes:

  mountOptions:
    - async
    - nfsvers=4.2
    - noatime

These do not appear to affect (or improve) the NFS performance at all in this case. Based on the other deployments I have utilizing NFS, this seems odd.

[thunerbl]

Hello there,

I've got the same issue, NFS PVC works well with Nextcloud v17.
But, as you @WojoInc with Nextcloud v19 i'm stuck at "Initializing Nextcloud...".

Even if installation seems fail, and the Pod loop on restart , My NFS volume seems wrotten by Nextcloud v19 data.
I'm trying now to get more verbosity about that.

Have a nice time :)

[Scizoo88]

Hi,

I faced the same problem. I logged in to the physical node and watched the docker logs. There I saw that Nextcloud tried to connect via HTTP to the defined Host. I have HAProxy (OPNSense) in front of Kubernetes and redirect all HTTP to HTTPS. This was an issue. For the init process of Nextcloud I temporary added the HTTP rule for it and the process completed without problems.

Maybe you have a similar setup?

BR
Scizoo

[thunerbl]

Hello @Scizoo88,

Thanks for sharing your experience. I don't think I've that setup, because my Nextcloud 19 pod, without NFS PVC for now, is accessible both via HTTP and HTTPS.

In my case, the unique difference between a working and not working setup, is that i've enabled data persistence (if I choose Nextcloud v19).
Persistence greatly worked on Nextcloud 17, with the same Kubernetes network setup tought

Have a nice day,

[thunerbl]

Okeii I've managed to connect with externalDB, Nextcloud 19 seems installed and functionnal pretty well, PVC enabled.
Maybe this error is SQLite related.

[chrisingenhaag]

Hi guys,
I already checked this. We´re using a fixed fsGroup for the apache and the nginx container. Because nextcloud copies files around via rsync on startup it relies on valid permissions to the volumes.

But in my case the user id and groups on my nfs client mount are different. My logs show permission denied errors.

I see two possible solutions:

• add a sidecar or possibility for generic sidecar containers to make somethink like chown -R ....
• try to use securityContext.fsGroupChangePolicy = Always (kubernetes 1.18 alpha)

For the moment I would tend to go for sidecar possibility so that you guys can handle volume permissions by yourself.

Best

[somerandow]

I seem to run in to errors with permissions even when the nfs mount is owned by www-data. I have tried manually editing the securityContext to set the fsGroupChangePolicy, and this didn't seem to resolve the issue either. I'll dive in a bit more and test out whether a side car or init container could set the permissions correctly.

[somerandow]

I seem to have resolved the performance issues around the use of NFS. Rsync was being forced to use synchronous writes due to NFS default behavior and how rsync checks for copied files. CP was slightly faster, but the real fix was enabling the async option on the NFS export (I had only been adding this option to the Persistent Volume), at least for the initial install. This took the time to initialize nextcloud down from >15 mins to just under 10 seconds.

I plan to test the permissions if the permissions are still an issue now.

[J3m5]

I'm experiencing the same problem, I tried to change the securityContext params but that didn't solve the problem...

[davad]

I think I'm having the same issue:

1. the container is being periodically restarted
2. the only output to the log is "Initializing nextcloud 19.0.3.1 ..."
3. the PVC is automatically created from my NFS storage class

I'll try adding the async option to the host and PV, then report back.

Edit: having trouble adding async to my NFS server because of the storage class provider I'm using.