Feature: multipath support

[ghormoon]

It would be very useful, if we could have anysort of multipath support, so we can connect to HA storages.

I'm willing to try to implement it, just want to discuss the design. my initial idea is to change the "target" variables to be under targets (with a fallback, that if they are set, they will be put into targets[0])

so this:

    volumeAttributes:
      targetTrAddr: "192.168.122.18"
      targetTrPort: "49153"
      targetTrType: "tcp"
      deviceUUID: "58668891-c3e4-45d0-b90e-824525c16080"
      nqn: "nqn.2022-08.org.test-nvmf.example"

would work the same as this:

    volumeAttributes:
      targets:
        - trAddr: "192.168.122.18"
          trPort: "49153"
          trType: "tcp"
      deviceUUID: "58668891-c3e4-45d0-b90e-824525c16080"
      nqn: "nqn.2022-08.org.test-nvmf.example"

with an option to add multiple

    volumeAttributes:
      targets:
        - trAddr: "192.168.122.18"
          trPort: "49153"
          trType: "tcp"
        - trAddr: "192.168.122.19"
          trPort: "49153"
          trType: "tcp"
      deviceUUID: "58668891-c3e4-45d0-b90e-824525c16080"
      nqn: "nqn.2022-08.org.test-nvmf.example"

I need to do a test, if the trType can be different per port (i.e. one rdma, one tcp), if not, it can stay on the top level and not be replicated in each block to not confuse users. I will confirm that later.

Another option how to implement multipath is to make the driver detect all the ports, same as nvme connect-all, but that has the downside that if the given ip is dead, it will not be able to detect. So I think if we want to enable that, we still want to support multiple IPs, so it can run the detection against all of them.

[ghormoon]

seems i cannot assign an issue to myself, can you do it please?

[MeinhardZhou]

I find this requirement somewhat challenging to implement because it conflicts with the CSI driver interface.

In the CSI driver, mounting a volume is achieved through the NodePublishVolumeRequest, which only supports a single volume ID. I may need to further investigate whether there are alternative approaches to mounting multiple disks for a pod.

[ghormoon]

I have a rough idea how to implement it and that happens at the point we're connecting to the nvme target, we don't need to put together the multipath IN the pod, but before we pass it in the pod.

to implement the most basic way, we only need to connect all the IPs that we are given, same as in this example i did manually (using the same hostid+hostnqn+nqn)

# nvme connect -t rdma -a 10.250.250.44 -s 4420 --nqn=nqn.2010-06.com.purestorage:flasharray.abcdefgh01234567 
connecting to device: nvme4

# nvme list-subsys
nvme-subsys3 - NQN=nqn.2010-06.com.purestorage:flasharray.abcdefgh01234567
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-aabbccddeeff
\
 +- nvme4 rdma traddr=10.250.250.44,trsvcid=4420 live

# ls -la /dev/disk/by-id/| grep nvme3
lrwxrwxrwx  1 root root  13 Oct 13 09:30 nvme-eui.00aabbccddeeff001122334455667788 -> ../../nvme3n3
lrwxrwxrwx  1 root root  13 Oct 13 09:30 nvme-eui.00aabbccddeeff001122334455667789 -> ../../nvme3n2
lrwxrwxrwx  1 root root  13 Oct 13 09:30 nvme-Pure_Storage_FlashArray_ABCDEFGH01234567_30 -> ../../nvme3n3
lrwxrwxrwx  1 root root  13 Oct 13 09:30 nvme-Pure_Storage_FlashArray_ABCDEFGH01234567_31 -> ../../nvme3n2

# nvme connect -t rdma -a 10.250.250.43 -s 4420 --nqn=nqn.2010-06.com.purestorage:flasharray.abcdefgh01234567 
connecting to device: nvme5

# nvme list-subsys
nvme-subsys3 - NQN=nqn.2010-06.com.purestorage:flasharray.abcdefgh01234567
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-aabbccddeeff
\
 +- nvme4 rdma traddr=10.250.250.44,trsvcid=4420 live
 +- nvme5 rdma traddr=10.250.250.43,trsvcid=4420 live

# ls -la /dev/disk/by-id/| grep nvme3
lrwxrwxrwx  1 root root  13 Oct 13 09:31 nvme-eui.00aabbccddeeff001122334455667788 -> ../../nvme3n3
lrwxrwxrwx  1 root root  13 Oct 13 09:31 nvme-eui.00aabbccddeeff001122334455667789 -> ../../nvme3n2
lrwxrwxrwx  1 root root  13 Oct 13 09:31 nvme-Pure_Storage_FlashArray_ABCDEFGH01234567_30 -> ../../nvme3n3
lrwxrwxrwx  1 root root  13 Oct 13 09:31 nvme-Pure_Storage_FlashArray_ABCDEFGH01234567_31 -> ../../nvme3n2

the amount of connected IPs does not have any effect on the resulting path we're later passing via symlink to the pod.
the nvme subsystem id doesn't change and the links in /dev/disk/by-id neither

so the only major change that needs to happen is that we're able to pass multiple IP addresses in the volume definition as i proposed in the original comment

I plan to implement some PoC either around end of the week or the next