Skip to content

Vol mount failed - iscsi: error logging in target: exit status 15 #520

@sunnygupta1218

Description

@sunnygupta1218

Issue description:
On rebooting node, pod with pvc comes in unknown state and throws following error:

Pod describe o/p:

Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  4m1s (x659 over 22h)  kubelet  MountVolume.MountDevice failed for volume "csi-b4786769-b306-40ac-9726-dc7aa313b780" : rpc error: code = Internal desc = iscsi: error logging in target: exit status 15

Environment details


CSI release =v1.33.0
K8 version = v1.35.0
OS release = "Ubuntu 24.04.3 LTS"

CSI Node error:

2026-02-06T10:34:54.428Z        ERROR   BV      driver/bv_node.go:192   failed to log into the iSCSI target.    {"volumeID": "ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza", "stagingPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount", "error": "iscsi: error logging in target: exit status 15"}
2026-02-06T10:34:54.428Z        ERROR   BV      driver/driver.go:352    Failed to process gRPC request. {"error": "rpc error: code = Internal desc = iscsi: error logging in target: exit status 15", "method": "/csi.v1.Node/NodeStageVolume", "request": "{\"publish_context\":{\"attachment-type\":\"iscsi\",\"device\":\"/dev/oracleoci/oraclevdb\",\"iscci_iqn\":\"iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909\",\"iscsi_ip\":\"169.254.2.3\",\"iscsi_port\":\"3260\",\"multipathDevices\":\"\",\"multipathEnabled\":\"false\",\"needResize\":\"false\",\"newSize\":\"\",\"vpusPerGB\":\"10\"},\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{}},\"volume_context\":{\"attachment-type\":\"\",\"needResize\":\"false\",\"newSize\":\"\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1770228548244-4488-blockvolume.csi.oraclecloud.com\",\"vpusPerGB\":\"10\"},\"volume_id\":\"ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza\"}"}
2026-02-06T10:34:57.802Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.803Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.804Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.806Z        DEBUG   FSS     driver/fss_node.go:62   volumeHandler :  &{ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa 10.0.0.8 /csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8}  {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.806Z        DEBUG   FSS     driver/fss_node.go:78   volume context: map[encryptInTransit:false storage.kubernetes.io/csiProvisionerIdentity:1770275984752-42-fss.csi.oraclecloud.com]       {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z        DEBUG   FSS     driver/fss_node.go:130  Trying to stage.        {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z        INFO    FSS     driver/fss_node.go:149  Stage started.  {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
--

At OS layer, it seems the device is already logged in however CSI Driver also tried to do and fails

ubuntu@instance-20260202-1239:/var/log$ sudo iscsiadm -m session 
tcp: [1] 169.254.0.2:3260,1 iqn.2015-02.oracle.boot:uefi (non-flash)
tcp: [2] 169.254.2.3:3260,1 iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909 (non-flash)

If I logout from session using iscsiadm command at OS layer and restart the pod, then volume is mounted successfully. This manual workaround is not feasible for production environment.

csi-oci-node.tar.gz

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions