nfs provisioner pod (nfs-provisioner-7cc5674bf7-mfdwd) entered into CrashLoopBackOff status and report the following logs

[Arjun-D7]

Hi Jooho,

Description of problem:
nfs provisioner pod (nfs-provisioner-7cc5674bf7-mfdwd) entered into CrashLoopBackOff status and report the following logs
Error setting up NFS server: error writing ganesha config /export/vfs.conf: open /export/vfs.conf: read-only file system

openshift 4.10.31
kubernetes v1.23.5+012e945

How reproducible:
always

Steps to Reproduce:

1. install NFS Provisioner Operator from the operators hub
2. create instance nfsprovisioner-sample from the operator

---

apiVersion: cache.jhouse.com/v1alpha1
kind: NFSProvisioner
metadata:
name: nfsprovisioner-sample
namespace: openshift-operators
spec:
nodeSelector:
app: nfs-provisioner
nfsImageConfiguration:
image: 'k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.1'
imagePullPolicy: IfNotPresent
scForNFS: nfs
hostPathDir: /shared

---

3, make hostpath requirements
4. create Service and Pod

Actual results:
Pod entered into "CrashLoopBackOff" status.

Expected results:
Pod can be run successfully.

PVC Dump:

oc get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
nfs-pvc-example Pending nfs 29m
[root@ocprovision shared]# oc describe pvc nfs-pvc-example
Name: nfs-pvc-example
Namespace: openshift-operators
StorageClass: nfs
Status: Pending
Volume:
Labels:
Annotations: volume.beta.kubernetes.io/storage-provisioner: example.com/nfs
volume.kubernetes.io/storage-provisioner: example.com/nfs
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By:
Events:
Type Reason Age From Message

---

Normal ExternalProvisioning 4m44s (x106 over 29m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "example.com/nfs" or manually created by system administrator

StorageClass Dump:

oc describe sc nfs

Name: nfs
IsDefaultClass: No
Annotations:
Provisioner: example.com/nfs
Parameters: mountOptions=vers=4.1
AllowVolumeExpansion:
MountOptions:
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events:

Additional info:

[root@worker1 ~]# df -h /shared
Filesystem Size Used Avail Use% Mounted on
10.72.33.153:/shared 54T 42T 12T 79% /shared
[root@worker1 ~]# ls -ldZ /shared
drwxrwxrwx. 76 root root system_u:object_r:container_file_t:s0 8192 Oct 10 12:33 /shared
[root@worker1 ~]#

[root@ocprovision shared]# oc get pods
NAME READY STATUS RESTARTS AGE
devworkspace-controller-manager-9bd98cc46-pm5qk 2/2 Running 0 5d20h
devworkspace-webhook-server-67c4c4d468-4tb7l 2/2 Running 0 5d20h
nfs-provisioner-7cc5674bf7-mfdwd 0/1 CrashLoopBackOff 9 (52s ago) 22m
nfs-provisioner-operator-controller-manager-58cd8448f-25h4j 2/2 Running 0 62m
web-terminal-controller-6dbbc96fc8-c2tbw 1/1 Running 2 (5d20h ago) 5d20h

oc describe pod nfs-provisioner-7cc5674bf7-mfdwd

Name: nfs-provisioner-7cc5674bf7-mfdwd
Namespace: openshift-operators
Priority: 0
Node: worker1.se01ocp001.internal.odencluster.com/10.72.73.21
Start Time: Thu, 13 Oct 2022 02:17:50 -0400
Labels: app=nfs-provisioner
nfsprovisioner_cr=nfsprovisioner-sample
pod-template-hash=7cc5674bf7
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.131.2.238"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.131.2.238"
],
"default": true,
"dns": {}
}]
openshift.io/scc: nfs-provisioner
Status: Running
IP: 10.131.2.238
IPs:
IP: 10.131.2.238
Controlled By: ReplicaSet/nfs-provisioner-7cc5674bf7
Containers:
nfs-provisioner:
Container ID: cri-o://04c04b6bcfef2d99cbfbe352e580c72fbee28c507884f97d7f38c6dc0494223d
Image: k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.1
Image ID: k8s.gcr.io/sig-storage/nfs-provisioner@sha256:cb28792f279836e31e7b4cbd1f617cad095fe2caef54e3f4aca66a8e4d8f7dcc
Ports: 2049/TCP, 2049/UDP, 32803/TCP, 32803/UDP, 20048/TCP, 20048/UDP, 875/TCP, 875/UDP, 111/TCP, 111/UDP, 662/TCP, 662/UDP
Host Ports: 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/UDP, 0/TCP, 0/UDP
Args:
'-provisioner=example.com/nfs'
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Thu, 13 Oct 2022 02:38:59 -0400
Finished: Thu, 13 Oct 2022 02:38:59 -0400
Ready: False
Restart Count: 9
Environment:
POD_IP: (v1:status.podIP)
SERVICE_NAME: nfs-provisioner
POD_NAMESPACE: openshift-operators (v1:metadata.namespace)
Mounts:
/export from export-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-chpbp (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
export-volume:
Type: HostPath (bare host directory volume)
Path: /shared
HostPathType: Directory
kube-api-access-chpbp:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional:
QoS Class: BestEffort
Node-Selectors: app=nfs-provisioner
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 22m default-scheduler Successfully assigned openshift-operators/nfs-provisioner-7cc5674bf7-mfdwd to worker1.se01ocp001.internal.odencluster.com
Normal AddedInterface 22m multus Add eth0 [10.131.2.238/23] from openshift-sdn
Normal Pulled 20m (x5 over 22m) kubelet Container image "k8s.gcr.io/sig-storage/nfs-provisioner:v3.0.1" already present on machine
Normal Created 20m (x5 over 22m) kubelet Created container nfs-provisioner
Normal Started 20m (x5 over 22m) kubelet Started container nfs-provisioner
Warning BackOff 2m5s (x93 over 22m) kubelet Back-off restarting failed container
[root@ocprovision shared]# oc logs -f nfs-provisioner-7cc5674bf7-mfdwd
I1013 06:38:59.460963 1 main.go:65] Provisioner example.com/nfs specified
I1013 06:38:59.461745 1 main.go:89] Setting up NFS server!
I1013 06:38:59.540906 1 server.go:149] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I1013 06:38:59.540930 1 server.go:160] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
F1013 06:38:59.540976 1 main.go:92] Error setting up NFS server: error writing ganesha config /export/vfs.conf: open /export/vfs.conf: read-only file system

SCC used:

oc get scc nfs-provisioner -o yaml

allowHostDirVolumePlugin: true
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities:

• DAC_READ_SEARCH
• SYS_RESOURCE
  apiVersion: security.openshift.io/v1
  defaultAddCapabilities: null
  fsGroup:
  type: MustRunAs
  groups: []
  kind: SecurityContextConstraints
  metadata:
  creationTimestamp: "2022-10-13T05:46:56Z"
  generation: 1
  name: nfs-provisioner
  ownerReferences:
  • apiVersion: cache.jhouse.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: NFSProvisioner
    name: nfsprovisioner-sample
    uid: 4fb03b1b-7422-44f1-b7fb-5737344e3eb3
    resourceVersion: "100451806"
    uid: e0365e0d-7123-45e7-ab5d-5f3a8f598cee
    priority: null
    readOnlyRootFilesystem: false
    requiredDropCapabilities:
• KILL
• MKNOD
• SYS_CHROOT
  runAsUser:
  type: RunAsAny
  seLinuxContext:
  type: MustRunAs
  supplementalGroups:
  type: RunAsAny
  users:
• system:serviceaccount:openshift-operators:nfs-provisioner
  volumes:
• configMap
• downwardAPI
• emptyDir
• hostPath
• persistentVolumeClaim
• secret

Regards,
Arjun D