Volume Busy, NodeStageVolume is already in progress #210
Description
What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)
Step1: Install iSCSI initiator and start service on every k8s node
sudo apt install open-iscsi
sudo systemctl enable --now iscsid
modprobe iscsi_tcp
echo iscsi_tcp >/etc/modules-load.d/iscsi-tcp.conf
systemctl start iscsid
Step2: Add extra_binds under kubelet service in cluster YAML
services:
kubelet:
extra_binds:
- "/etc/iscsi:/etc/iscsi"
- "/sbin/iscsiadm:/sbin/iscsiadm"
- "/var/lib/iscsi:/var/lib/iscsi"
- "/var/openebs/local:/var/openebs/local"
- "/lib/modules"
Step3: install openebs jiva with helm3 follow the Quickstart
helm repo add openebs-jiva https://openebs.github.io/jiva-operator
helm repo update
helm install jiva openebs-jiva/jiva --namespace openebs --create-namespace
Step4: Confirm all of openebs pod is running
Step5: Create a pvc with the sc openebs-jiva-csi-default and all the pvc are bound state
Step6: Create a deployment use the pvc
What did you expect to happen:
I expect everything is ok but is not ok.
Events:
Type Reason Age From Message
Normal Scheduled 42m default-scheduler Successfully assigned kafka/nginx-b9cd4c87-8k4sx to rke-node-219
Warning FailedMount 40m (x3 over 40m) kubelet MountVolume.MountDevice failed for volume "pvc-ba7732f8-58cb-4dd2-a682-ec4ca2cac7dc" : rpc error: code = Aborted desc = Volume Busy, NodeStageVolume is already in progress
Warning FailedMount 13m (x5 over 29m) kubelet Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[default-token-xtmdg vol1]: timed out waiting for the condition
Warning FailedMount 4m13s (x12 over 40m) kubelet Unable to attach or mount volumes: unmounted volumes=[vol1], unattached volumes=[vol1 default-token-xtmdg]: timed out waiting for the condition
Warning FailedMount 4s (x13 over 40m) kubelet MountVolume.MountDevice failed for volume "pvc-ba7732f8-58cb-4dd2-a682-ec4ca2cac7dc" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)
kubectl logs <jiva-operator pod name> -n openebs(optional)kubectl get jv <jiva volume cr name> -n openebs -o yamlkubectl get jvp <jiva volume policy> -n openebs -o yaml
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
- Jiva version 3.4.0
- OpenEBS version 3.4.0
- Kubernetes version (use
kubectl version):
Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-04-14T13:21:19Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.15", GitCommit:"8f1e5bf0b9729a899b8df86249b56e2c74aebc55", GitTreeState:"clean", BuildDate:"2022-01-19T17:23:01Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} - Kubernetes installer & version:rancher2.5.14
- Cloud provider or hardware configuration:
- OS (e.g. from
/etc/os-release):ubuntu22.04