-
Notifications
You must be signed in to change notification settings - Fork 100
Open
Description
Issue description:
On rebooting node, pod with pvc comes in unknown state and throws following error:
Pod describe o/p:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 4m1s (x659 over 22h) kubelet MountVolume.MountDevice failed for volume "csi-b4786769-b306-40ac-9726-dc7aa313b780" : rpc error: code = Internal desc = iscsi: error logging in target: exit status 15
Environment details
CSI release =v1.33.0
K8 version = v1.35.0
OS release = "Ubuntu 24.04.3 LTS"
CSI Node error:
2026-02-06T10:34:54.428Z ERROR BV driver/bv_node.go:192 failed to log into the iSCSI target. {"volumeID": "ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza", "stagingPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount", "error": "iscsi: error logging in target: exit status 15"}
2026-02-06T10:34:54.428Z ERROR BV driver/driver.go:352 Failed to process gRPC request. {"error": "rpc error: code = Internal desc = iscsi: error logging in target: exit status 15", "method": "/csi.v1.Node/NodeStageVolume", "request": "{\"publish_context\":{\"attachment-type\":\"iscsi\",\"device\":\"/dev/oracleoci/oraclevdb\",\"iscci_iqn\":\"iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909\",\"iscsi_ip\":\"169.254.2.3\",\"iscsi_port\":\"3260\",\"multipathDevices\":\"\",\"multipathEnabled\":\"false\",\"needResize\":\"false\",\"newSize\":\"\",\"vpusPerGB\":\"10\"},\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{}},\"volume_context\":{\"attachment-type\":\"\",\"needResize\":\"false\",\"newSize\":\"\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1770228548244-4488-blockvolume.csi.oraclecloud.com\",\"vpusPerGB\":\"10\"},\"volume_id\":\"ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza\"}"}
2026-02-06T10:34:57.802Z INFO FSS driver/driver.go:354 gRPC response is sent successfully. {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.803Z INFO FSS driver/driver.go:354 gRPC response is sent successfully. {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.804Z INFO FSS driver/driver.go:354 gRPC response is sent successfully. {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.806Z DEBUG FSS driver/fss_node.go:62 volumeHandler : &{ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa 10.0.0.8 /csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8} {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.806Z DEBUG FSS driver/fss_node.go:78 volume context: map[encryptInTransit:false storage.kubernetes.io/csiProvisionerIdentity:1770275984752-42-fss.csi.oraclecloud.com] {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z DEBUG FSS driver/fss_node.go:130 Trying to stage. {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z INFO FSS driver/fss_node.go:149 Stage started. {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
--
At OS layer, it seems the device is already logged in however CSI Driver also tried to do and fails
ubuntu@instance-20260202-1239:/var/log$ sudo iscsiadm -m session
tcp: [1] 169.254.0.2:3260,1 iqn.2015-02.oracle.boot:uefi (non-flash)
tcp: [2] 169.254.2.3:3260,1 iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909 (non-flash)
If I logout from session using iscsiadm command at OS layer and restart the pod, then volume is mounted successfully. This manual workaround is not feasible for production environment.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels