Add support for NVMe-oF volumes backed by RBD

[nixpanic]

Describe the feature you'd like to have

It would be great to have a Ceph-CSI provisioner for creating RBD volumes and have them accessible over NVMe-oF.

What is the value to the end user? (why is it a priority?)

Users that can not use native RBD-clients (like MS Windows hosts) can still use Ceph RBD.

How will we know we have a good solution? (acceptance criteria)

• Ceph-CSI should be able to create and delete RBD-images that are accessible over NVMe-oF
• simple deployment files under deploy/ are added
• e2e testing should deploy the Ceph nvme-gateway and run e2e (e2e: add NVMe-oF gateway deployment #5614)
• integrate deploying the NVMe-oF controller- and node-plugin with ceph-csi-operator (Add NVMe-oF driver type ceph-csi-operator#375)

Additional context

• Rook needs to deploy the NVMe-oF gateway docs: add design doc for ceph nvmeof gateway rook/rook#16020
• There needs to be a tool or API included in the Ceph base container image for configuring (doc)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[gadididi]

the first nvmeof-csi PR : #5452
provides the provisioner.
there are some known issue \ open tasks that must be fixed in the next PR:

• Change the create\delete order: create subsystem -> listener -> ns -> add host (opposite direction for deletion)
• If some resource exists, don’t raise failure (need to discuss which resources)
• Add more listeners to sc (temporary solution until the gateway does it automatically)
• Add more GW's IP to sc
• Check ceph manager to get IPs list? the gateway will handle listeners in the future
• Create util-nvmeof package and move util function to nvmeof_util.go file
• Makes create\Delete calls fullly idempotent
• cleanup resources when CreateVolume fails (nvmeof: cleanup resources when CreateVolume fails #5538)
• Add ns UUID into NVMeoFVolumeData , for NodeStage
• Add support for RADOS namespace ( pool/namespace/rbd_image_name) (Add the option to create RADOS namespace ceph-nvmeof#1458)
• remove the generated gRPC/protobuf files (once Provide a Go module for interacting with the gateway ceph-nvmeof#1449 is merged)
• replace hostNQN with an allowAllHosts: (bool) parameter (when set to false, add hosts in ControllerPublishVolume) generate hostnqn in controller, and send it via VolumeContext to node-plugin. use --hostnqn= in nvme connect command UPDATE: update NodeID() embbeded method in nodeplugin to return nodeid+hostnqn, then controllerpublishvolume can use it.
• make listener parameters in StorageClass optional, use all available gateways (check with ceph nvme-gw show ...)
• verify RBD parameters in the StorageClass, report conflicting/unsupported parameters
• track/reference count subsystem usage in the Node-plugin, to call nvme disconnect when no NVMe volumes are attached anymore (JSON file with details on a hostPath)
• add nvme operation locking (prevents concurrent operations in case something is slow), like RBD has:
  

  ceph-csi/internal/rbd/controllerserver.go

  Lines 384 to 389 in 709f85e

  	if acquired := cs.VolumeLocks.TryAcquire(req.GetName()); !acquired {
  	log.ErrorLog(ctx, util.VolumeOperationAlreadyExistsFmt, req.GetName())
  	return nil, status.Errorf(codes.Aborted, util.VolumeOperationAlreadyExistsFmt, req.GetName())
  	}
  	defer cs.VolumeLocks.Release(req.GetName())

• unit tests for functions that do not communicate with the Ceph cluster
• add write\read lock mechanism to the controller and the node plugin