Typo in RBD modprobe (1.32/edge)

[QthePirate]

Ubuntu 24.04.2
ceph-csi 1.32/edge
k8s 1.32/stable
k8s-worker 1.32/stable

In deployment of a multi-architecture k8s cluster (ARM64 and AMD64) I couldn't determine why the csi-rbdplugin pod kept failing on the AMD64 k8s-worker node.

Turns out there is a typo when running modprobe:

W0615 20:39:57.845803 18942 rbd_attach.go:225] nbd modprobe failed (an error (exit status 1) occurred while running modprobe args: [nbd]): "modprobe: ERROR: could not insert 'nbd': Exec format error\n

It's trying to modprobe "nbd" not "rbd"

I manually ran modprobe rbd on the node and the pod worked properly.

[addyess]

@QthePirate is there any chance you spotted where the charm has this typo?

[addyess]

it looks to me the ceph-csi workload is responsible for loading this kernel module . And it seems the name nbd is intentional.

https://github.com/ceph/ceph-csi/blob/0778a37bcf79fe5e3a883526a89825ab01c88b11/internal/rbd/rbd_attach.go#L229-L245

I believe to use ceph-csi on noble, you must use a more recent release -- can you try:

juju config ceph-csi release=v3.13.0

rather than the default v3.9.0

[QthePirate]

@addyess
Is nbd intentional? I moved the version to v3.12.0 and it was the same error.
Correct, this is ceph-csi on noble. I'll try with 3.13 and see if the same error occurs.

[addyess]

Yes, i think nbd is intentional.... I think v3.13.0 is the first ceph-csi release that works on noble's kernel

[addyess]

see https://github.com/ceph/ceph-csi/blob/devel/internal/rbd/rbd_attach.go#L39

[QthePirate]

@addyess

I just verified, I was running on v3.14.0

[QthePirate]

Ok. So it's clear one of these modules is not loading properly. It did load fine on the ARM64 devices (noble as well) but not in the AMD64. But, it was working previously.

The only other difference here is the amd64 box is loading the k8s-worker charm and not the k8s charm with which ceph-csi subordinate charm is loaded.

Going to try a few things to see where the gap is either in documentation, the pod/container, or just not taken into account in a K8s multi-arch environment.

[addyess]

@QthePirate would you please confirm v3.13.0 not v3.13.0? The charm doesn't have support yet for 3.14.0

Also the ceph-csi charm should load the correct image on either arm64 or amd64. I believe the images support multi arch but I'm not 100%

[QthePirate]

Will adjust and attempt.

…

________________________________ From: Adam Dyess ***@***.***> Sent: Monday, June 16, 2025 4:01:47 PM To: charmed-kubernetes/ceph-csi-operator ***@***.***> Cc: Michael Quiniola ***@***.***>; Mention ***@***.***> Subject: Re: [charmed-kubernetes/ceph-csi-operator] Typo in RBD modprobe (1.32/edge) (Issue #49) [https://avatars.githubusercontent.com/u/10090033?s=20&v=4]addyess left a comment (charmed-kubernetes/ceph-csi-operator#49)<#49 (comment)> @QthePirate<https://github.com/QthePirate> would you please confirm v3.13.0 not v3.13.0? The charm doesn't have support yet for 3.14.0 Also the ceph-csi charm should load the correct image on either arm64 or amd64. I believe the images support multi arch but I'm not 100% — Reply to this email directly, view it on GitHub<#49 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJZQEJYCVFUWYS26EIXOBPL3D5EFXAVCNFSM6AAAAAB7LV6L7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZYGQZDOMRSGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[WizardBit]

In my testing setting the release to v3.13.0 on deployment did resolve the issue for me.

juju deploy ceph-csi --config provisioner-replicas=1 \ 
--config namespace="${CEPH_NS}" --config create-namespace=true --config release=v3.13.0

NAME                                                        PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/ceph-ext4                       rbd.csi.ceph.com         Delete          Immediate              true                   108s
storageclass.storage.k8s.io/ceph-xfs (default)              rbd.csi.ceph.com         Delete          Immediate              true                   108s
storageclass.storage.k8s.io/csi-rawfile-default (default)   rawfile.csi.openebs.io   Delete          WaitForFirstConsumer   true                   13m

NAME                                             READY   STATUS    RESTARTS   AGE
pod/csi-rbdplugin-86h74                          2/2     Running   0          108s
pod/csi-rbdplugin-dbz7r                          2/2     Running   0          108s
pod/csi-rbdplugin-kx6c4                          2/2     Running   0          108s
pod/csi-rbdplugin-provisioner-6f5fb97894-kpbs5   6/6     Running   0          108s