proc entry still exists during unmount

[everythings-gonna-be-alright]

Describe the bug
When you have a Kubernetes cluster with an application that mounts the host directory /var/lib in read-only mode, it leads to problems with the Mayastor remount process. The Mayastor CSI driver can't unmount the disk because the operating system sees that some processes still do not allow unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/{volume_id}/globalmount, even if the globalmount looks like unmounted, because the directory is empty, but the ext4 journaling process still exists, and the NVMe device still exists.
If you will delete the /var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/{volume_id} directory manually with the rm -r command, after this, the ext4 journaling process will stop, and the CSI driver can continue unmounting.

To Reproduce

1. Install the Vector log aggregator application into the Kubernetes cluster via Helm chart:
   ( By default, Vector will mount /var/lib for some reason and perhaps will try to read the entry )
   https://github.com/vectordotdev/helm-charts/blob/23f60fec2332b20a301796c80bf7c5b49b383045/charts/vector/values.yaml#L353

2. Try to create pods with volumes and try to restart them. For example, 30 pods with volumes for three times.
   Some parts of the pods ( like 20% or more ) will be stuck in the pending phase, and you will see the error proc entry still exists in the CSI driver logs.

Script to find empty directories on the node:

find /var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor -type d -name "globalmount" -empty -exec dirname {} \;

Script to find out which process is still trying to use those empty directories by NVMe device ID:

DEVICE='nvme0n1'
for pid in $(ls /proc | grep '^[0-9]\+$'); do
  if [ -r /proc/$pid/mounts ]; then
    if grep -q ${DEVICE} /proc/$pid/mounts; then
      echo "Mounted in PID $pid:"
      grep ${DEVICE} /proc/$pid/mounts
    fi
  fi
done

Expected behaviour
It looks like understandable behaviour, but it can be a bit tricky to investigate. + If we talk, for example, about the AWS CSI driver for EBS volumes, I can't reproduce this kind of behaviour on AWS. So I'm not sure, but maybe it is possible to fix it at the Mayastor code level to mitigate behaviour like this.

OS info:

• Distro: Ubuntu 24.04
• Kernel version: 6.14.0-29-generic
• MayaStor revision or container image: 2.9.2

[tiagolobocastro]

So the lockup happens when a pod is deleted, but another pod still has /var/lib mounted?

I remember that previously we had some issues disconnecting devices gracefully and used some dirty workaround involving waiting for the journal process to complete or something.

[everythings-gonna-be-alright]

@tiagolobocastro Yes. In this case, the pod with the application Vector mounts the entire host /var/lib directory in read-only mode. It works like a daemonset on each node and aggregates logs (like Logstash).

Meantime, I have, for example, a pod with MySQL on this node with a mounted MySQL-data volume in:
/var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/57d087e9f45f2fe609265234dadf1524d937e66aedeb24364fb298ca31715c1f/globalmount
and
/var/lib/kubelet/pods/01005282-6b71-406a-bfae-e9c9ac9f5fdc/volumes/kubernetes.io~csi/pvc-47325edd-f26d-472e-bce2-5d86b2a773b5/mount

Next, when I try to recreate the MySQL pod, the Mayastor CSI driver will successfully remove /var/lib/kubelet/pods/01005282-6b71-406a-bfae-e9c9ac9f5fdc/volumes/kubernetes.io~csi/pvc-47325edd-f26d-472e-bce2-5d86b2a773b5/mount directory, but then will fall into the proc entry still exists error. Only after manual removal of this already empty for some reason directory: /var/lib/kubelet/plugins/kubernetes.io/csi/io.openebs.csi-mayastor/57d087e9f45f2fe609265234dadf1524d937e66aedeb24364fb298ca31715c1f/globalmount, this error will disappear, and the CSI driver will unmount the device from the VM.

After disabling the /var/lib/ mounting in vector config, Mayastor works fine, and I can't reproduce the proc entry still exists error anymore.

[tiagolobocastro]

Yeah I can reproduce this quite easily with a plain pod mounting /var/lib

The globalmount disappears from the host, but it's still present in the "plain pod" mount if I exec into it.

[tiagolobocastro]

If we delete the globalmount on NodeUnstage we can get around this but I don't think this is a good idea, since it's the COs responsibility to handle the globalmount.
Anyhow, I've asked on sig-storage for some help regarding this.

An interim solution is to set HostToContainer on the /var/lib mount, ex:

      volumeMounts:
        - mountPath: "/var/lib"
          name: var-lib
          mountPropagation: HostToContainer

This prevents the mount from being propagated to the host