nvmeof: add nvmeof provisioner ability

[gadididi]

Describe what this PR does

This PR introduces NVMe-oF CSI driver support for Ceph-CSI, enabling RBD volumes to be exposed through NVMe-oF protocol instead of traditional kernel RBD mapping.

Components Added:

• NVMe-oF Controller Server: Complete wrapper around RBD CSI that orchestrates NVMe-oF resource creation and cleanup
• NVMe-oF Gateway Client: Full gRPC client implementation for managing NVMe-oF subsystems, namespaces, listeners, and hosts
• Clean Domain Architecture: Separate structs for CSI operations vs protobuf wire protocol with proper abstraction layers
• Comprehensive Metadata Management: Stores all NVMe-oF metadata (subsystem NQN, namespace ID, host NQN, listener details, gateway addresses) in RBD image metadata for reliable cleanup
• Integration Tests: Docker-based tests for validating NVMe-oF gateway operations with configurable environment

Complete Workflow:

CreateVolume:

1. Validate request with comprehensive parameter validation
2. Create RBD volume through backend CSI driver
3. Setup NVMe-oF resources:
   • Create/ensure subsystem exists
   • Create namespace in subsystem
   • Create listeners on specified addresses
   • Add host to subsystem
4. Store metadata in both VolumeContext (for NodeServer) and RBD metadata (for cleanup)

DeleteVolume:

1. Retrieve stored metadata from RBD image
2. Cleanup NVMe-oF resources in proper order:
   • Remove host from subsystem
   • Delete namespace
   • Delete listeners
   • Delete subsystem (if empty)
3. Delete RBD volume

Is there anything that requires special attention

⚠️ Prototype Status - Controller Operations Only ⚠️

This is a working prototype implementing the controller-side operations. The purpose of this PR is to:

• Demonstrate complete NVMe-oF CSI controller integration
• Validate the clean architecture approach with proper separation of concerns
• Get feedback on metadata management and cleanup strategies

What's Working:

• ✅ Complete CreateVolume/DeleteVolume operations with full NVMe-oF setup
• ✅ Production-ready NVMe-oF Gateway integration (subsystem, namespace, listener, host management)
• ✅ Robust metadata storage/retrieval with dual storage (VolumeContext + RBD metadata)
• ✅ Clean separation of management vs data plane (gateway addresses vs listener addresses)
• ✅ Integration tests with configurable environment variables
• ✅ Proper resource lifecycle management with idempotent operations

What's Missing/TODO:

• ❌ NodeServer implementation (NodeStageVolume, NodePublishVolume, NodeUnpublishVolume)
• ❌ NVMe-oF initiator operations (nvme connect/disconnect on nodes)
• ❌ Multi-gateway support (HA)
• ❌ Multi-listeners support
• ❌ MTLS Grpc
• ❌ **inband-authentication **
• ❌ Advanced features (volume expansion?, snapshots?, cloning?)

Key Design Decisions Needing Review:

• Architecture Pattern: Controller-only setup vs full CSI implementation
• Metadata Strategy: Dual storage (VolumeContext + RBD metadata) for reliability

Configuration Model:

StorageClass parameters support flexible deployment scenarios:

parameters:
  subsystemNQN: "nqn.2016-06.io.ceph:subsystem.production"
  hostNQN: "nqn.2016-06.io.ceph:host.k8s-node1"
  nvmeofGatewayAddress: "10.242.64.100"    # Management network
  nvmeofGatewayPort: "5500"                # gRPC port
  listenerIpAddress: "10.242.64.32"       # Data network  
  listenerPort: "4420"                    # NVMe-oF port
  listenerHostname: "ceph-nvmeof-gw1"

Related issues\PR\discussions

1. Add support for NVMe-oF volumes backed by RBD #5370
2. https://github.com/ceph/ceph-csi/blob/devel/docs/design/proposals/nvme-of.md

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

• Commit Message Formatting: Commit titles and messages follow
  guidelines in the developer
  guide.
• Reviewed the developer guide on Submitting a Pull
  Request
• Pending release
  notes
  updated with breaking and/or notable changes for the next major release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

---

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

• /retest ci/centos/<job-name>: retest the <job-name> after unrelated
  failure (please report the failure too!)

[nixpanic]

No major changes needed from my point of view. There are quite a few TODO's listed, those are all improvements we can work out later.

For an initial proof-of-concept this looks good to me. With your additional work-in-progress branch(es) I have been able to get this up and running 🥳

$ kubectl describe pvc
Name:          nvmeof-test-pvc
Namespace:     default
StorageClass:  ocs-storagecluster-ceph-nvmeof
Status:        Bound
Volume:        pvc-3a0994d7-6457-4966-977e-dc756b871af7
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nvmeof.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: nvmeof.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                 Age   From                                                                                                    Message
  ----    ------                 ----  ----                                                                                                    -------
  Normal  Provisioning           9s    nvmeof.csi.ceph.com_csi-nvmeofplugin-provisioner-745f6d9947-f56rd_3479ebb6-df20-477d-a7bb-737129b80352  External provisioner is provisioning volume for claim "default/nvmeof-test-pvc"
  Normal  ExternalProvisioning   9s    persistentvolume-controller                                                                             Waiting for a volume to be created either by the external provisioner 'nvmeof.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  ProvisioningSucceeded  8s    nvmeof.csi.ceph.com_csi-nvmeofplugin-provisioner-745f6d9947-f56rd_3479ebb6-df20-477d-a7bb-737129b80352  Successfully provisioned volume pvc-3a0994d7-6457-4966-977e-dc756b871af7

[yati1998]

I went through the code, looks good to me from the demo we had. The missing parts already have TODO tag in it, so we are good to have them in upcoming PR.

[Madhu-1]

@nixpanic the Plan is to get this PR merged and later remove the pb files and integration test file when ceph PR is merged?

[Madhu-1]

Sounds good, its good to have single common function for connect

[Madhu-1]

Any plan to open a tracker to improve the documentation for consumers? or else it will become hard to understand the input and outputs

[Madhu-1]

it would be nice to return error early and avoid multiple if if we check resp.GetStatus() != 0 && resp.GetStatus() != int32(syscall.ENOENT) same is required in other Delete calls as we..

[gadididi]

@Madhu-1 , I can do it , but i want a specific check to handle "no exist" case.

what I can do is something like this:

	resp, err := gw.client.DeleteListener(ctx, req)
	if err != nil {
		return fmt.Errorf("failed to delete listener %s from subsystem %s: %w", listenerInfo.Address, subsystemNQN, err)
	}

	// Early return for error cases (anything other than success or not-found)
	if resp.GetStatus() != 0 && resp.GetStatus() != int32(syscall.ENOENT) {
		return fmt.Errorf("gateway DeleteListener returned error (status=%d): %s", resp.GetStatus(), resp.GetErrorMessage())
	}

	// Not-found cases
    if resp.GetStatus() == int32(syscall.ENOENT) {
		log.DebugLog(ctx, "Listener %s already deleted from subsystem %s (not found)", listenerInfo.Address, subsystemNQN)
         return nil
	}
   // Handle success 
   log.DebugLog(ctx, "Listener deleted successfully: %s from subsystem %s", listenerInfo.Address, subsystemNQN)
	return nil

but maybe it makes it more complicated, what do you think?

[Madhu-1]

i assume this is for logging, IMO we dont need to log if it is success or NotFound because its same for CSI where the deletion is successful. if really required we can log the status code at the end which helps for debugging. its not worth to add a if statement to log different error messages.

[gadididi]

I'd like to keep the distinction because logging "deleted successfully" when it was already gone feels inaccurate. The extra if helps with debugging to know if we actually deleted something vs. it was already missing.

[Madhu-1]

@nixpanic the Plan is to get this PR merged and later remove the pb files and integration test file when ceph PR is merged?

Based on the response, will LGTM, no need to address the new review comments, if we are planning to merge it. it can be a followup PR

[gadididi]

Hi, @Madhu-1 , @nixpanic ,

I pushed 2 new commits about generated files.

Update: Using upstream protobuf files from ceph-nvmeof

I've updated the PR to use the generated protobuf files from upstream ceph-nvmeof instead of maintaining local copies in our repository.

What changed:

• Added dependency on github.com/ceph/ceph-nvmeof/lib/go/nvmeof which now contains the generated gateway.pb.go and gateway_grpc.pb.go files
• Removed our locally generated protobuf files
• Vendored the upstream files into vendor/github.com/ceph/ceph-nvmeof/lib/go/nvmeof/
• Updated imports to use the upstream package: import pb "github.com/ceph/ceph-nvmeof/lib/go/nvmeof"

Commands I ran:

go get github.com/ceph/ceph-nvmeof@devel
go mod tidy
go mod vendor

Note about Go version change:
The Go version in go.mod was automatically updated from 1.24.0 to 1.24.4 because the upstream ceph-nvmeof module requires Go 1.24.4

[Madhu-1]

LGTM, Lets get it merged to devel once we have 3.15.0 release

[Madhu-1]

Added DNM to avoid accidental merge, please remove it once we have new release

[Madhu-1]

Thank you @gadididi for all the work 🎉

[Madhu-1]

i would suggest to drop 24951fb commit as its removing the proto file from internal folder and remove pb file from 1c430a5 this commit

[gadididi]

thank you! 🙂
yes, Niels told me wait until will be new release
I will do rebase interactive and rearrange the commits (squash 24951fb and 1c430a5)

Pull request has been modified.

[nixpanic]

@Mergifyio rebase

release-v3.15 has been branched, this can now get merged in the devel branch

[mergify]

rebase

☑️ Nothing to do, the required conditions are not met

Details

• queue-position = -1 [📌 rebase requirement]
• -closed [📌 rebase requirement]
• -conflict [📌 rebase requirement]
• any of:
  • #commits > 1 [📌 rebase requirement]
  • #commits-behind > 0 [📌 rebase requirement]
  • -linear-history [📌 rebase requirement]