rbd: use volumeattributesclass feature implement rbd volume qos by YiteGu · Pull Request #6160 · ceph/ceph-csi

YiteGu · 2026-03-06T09:34:06Z

Describe what this PR does

The solution using storageclass to implement RBD volume QoS has been cancelled because the storageclass parameters cannot be modified. The volumeattributesclass solution is more flexible because it supports mutable parameters.
Add the new parameter: volumeAttributesClassName when defining the PVC. For example:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vac-pvc-1
  namespace: rook-ceph
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
  storageClassName: rook-ceph-block
  volumeMode: Block
  volumeAttributesClassName: silver

Implemented the ControllerModifyVolume interface. Users can upgrade their QoS by modifying the volumeAttributesClassName parameter of an existing PVC.

ref: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

Is there anything that requires special attention

Do you have any questions?

Is the change backward compatible?

Are there concerns around backward compatibility?

Provide any external context for the change, if any.

For example:

Kubernetes links that explain why the change is required
CSI spec related changes/catch-up that necessitates this patch
golang related practices that necessitates this change

Related issues

Mention any github issues relevant to this PR. Adding below line
will help to auto close the issue once the PR is merged.

Fixes: #issue_number

Future concerns

List items that are not part of the PR and do not impact it's
functionality, but are work items that can be taken up subsequently.

Checklist:

Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
Reviewed the developer guide on Submitting a Pull
Request
Pending release
notes
updated with breaking and/or notable changes for the next major release.
Documentation has been updated, if necessary.
Unit tests have been added, if necessary.
Integration tests have been added, if necessary.

Show available bot commands

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

nixpanic

This is a user visible change. Put a note in PendingReleaseNotes.md for this too. Maybe add a note in the main README.md for support of this as well, update other documentation?

internal/rbd/controllerserver.go

e2e/rbd.go

internal/util/idlocker.go

mergify · 2026-03-10T16:43:22Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

mergify · 2026-03-11T05:14:21Z

This pull request now has conflicts with the target branch. Could you please resolve conflicts and force push the corrected changes? 🙏

nixpanic

Two minor things that caught the eye of Claude Code, and look like good recommendations to me.

internal/rbd/controllerserver.go

examples/rbd/storageclass.yaml

YiteGu · 2026-03-11T11:21:33Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic

Another round of review (now completed by me, not Claude).

These are the only changes I would like to see, everything else looks good to me. Thanks!

PendingReleaseNotes.md

README.md

docs/volumeattributesclass.md

nixpanic

Thanks, looks good to me.

internal/rbd/controllerserver.go

nixpanic · 2026-03-30T15:49:08Z

Need to run @mergify rebase, wait until the rebase is done, and add the ok-to-test label after #6198 is merged.

iPraveenParihar · 2026-03-31T04:43:17Z

@Mergifyio rebase

mergify · 2026-03-31T04:43:30Z

rebase

✅ Branch has been successfully rebased

ceph-csi-bot · 2026-03-31T04:57:21Z

/test ci/centos/k8s-e2e-external-storage/1.33

ceph-csi-bot · 2026-03-31T04:57:21Z

/test ci/centos/k8s-e2e-external-storage/1.34

ceph-csi-bot · 2026-03-31T04:57:21Z

/test ci/centos/mini-e2e-helm/k8s-1.33

ceph-csi-bot · 2026-03-31T04:57:21Z

/test ci/centos/upgrade-tests-cephfs

ceph-csi-bot · 2026-03-31T04:57:22Z

/test ci/centos/mini-e2e-helm/k8s-1.34

ceph-csi-bot · 2026-03-31T04:57:22Z

/test ci/centos/mini-e2e/k8s-1.33

ceph-csi-bot · 2026-03-31T04:57:22Z

/test ci/centos/k8s-e2e-external-storage/1.35

ceph-csi-bot · 2026-03-31T04:57:22Z

/test ci/centos/upgrade-tests-rbd

ceph-csi-bot · 2026-03-31T04:57:22Z

/test ci/centos/mini-e2e/k8s-1.34

ceph-csi-bot · 2026-03-31T04:57:23Z

/test ci/centos/mini-e2e-helm/k8s-1.35

ceph-csi-bot · 2026-03-31T04:57:24Z

/test ci/centos/mini-e2e/k8s-1.35

nixpanic · 2026-03-31T06:59:47Z

/retest ci/centos/upgrade-tests-rbd

nixpanic · 2026-03-31T07:00:21Z

/retest ci/centos/upgrade-tests-rbd

1st run failed with context deadline exceeded.

iPraveenParihar · 2026-03-31T08:37:06Z

@YiteGu, helm tests are failing with

  I0331 06:59:22.580179 80372 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - volumeattributesclasses.storage.k8s.io "silver" is forbidden: User "system:serviceaccount:cephcsi-e2e-6a8d7a61a22b:csi-rbdplugin-provisioner" cannot get resource "volumeattributesclasses" in API group "storage.k8s.io" at the cluster scope
  I0331 06:59:24.574928 80372 pvc.go:63] waiting for PVC rbd-pvc (600 seconds elapsed)
  I0331 06:59:24.575056 80372 pvc.go:66] Error getting pvc "rbd-pvc" in namespace "rbd-3886": client rate limiter Wait returned an error: context deadline exceeded
  I0331 06:59:24.575086 80372 utils.go:2046] failed to create PVC and application: failed to get pvc: client rate limiter Wait returned an error: context deadline exceeded

can you check on this?

nixpanic · 2026-03-31T11:12:08Z

@YiteGu, helm tests are failing with

  I0331 06:59:22.580179 80372 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - volumeattributesclasses.storage.k8s.io "silver" is forbidden: User "system:serviceaccount:cephcsi-e2e-6a8d7a61a22b:csi-rbdplugin-provisioner" cannot get resource "volumeattributesclasses" in API group "storage.k8s.io" at the cluster scope
  I0331 06:59:24.574928 80372 pvc.go:63] waiting for PVC rbd-pvc (600 seconds elapsed)
  I0331 06:59:24.575056 80372 pvc.go:66] Error getting pvc "rbd-pvc" in namespace "rbd-3886": client rate limiter Wait returned an error: context deadline exceeded
  I0331 06:59:24.575086 80372 utils.go:2046] failed to create PVC and application: failed to get pvc: client rate limiter Wait returned an error: context deadline exceeded

can you check on this?

I guess the Helm Chart creates the RBAC without access to the volumeattributesclass APIs.
Probably needs to be added to https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/provisioner-clusterrole.yaml

Pull request has been modified.

YiteGu · 2026-04-01T02:42:12Z

@YiteGu, helm tests are failing with

  I0331 06:59:22.580179 80372 pvc.go:85] PVC rbd-pvc Event: ProvisioningFailed - volumeattributesclasses.storage.k8s.io "silver" is forbidden: User "system:serviceaccount:cephcsi-e2e-6a8d7a61a22b:csi-rbdplugin-provisioner" cannot get resource "volumeattributesclasses" in API group "storage.k8s.io" at the cluster scope
  I0331 06:59:24.574928 80372 pvc.go:63] waiting for PVC rbd-pvc (600 seconds elapsed)
  I0331 06:59:24.575056 80372 pvc.go:66] Error getting pvc "rbd-pvc" in namespace "rbd-3886": client rate limiter Wait returned an error: context deadline exceeded
  I0331 06:59:24.575086 80372 utils.go:2046] failed to create PVC and application: failed to get pvc: client rate limiter Wait returned an error: context deadline exceeded

can you check on this?

I guess the Helm Chart creates the RBAC without access to the volumeattributesclass APIs. Probably needs to be added to https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/provisioner-clusterrole.yaml

updated

YiteGu · 2026-04-01T02:43:15Z

/test ci/centos/mini-e2e-helm/k8s-1.34

Signed-off-by: Yite Gu <guyite@bytedance.com>

YiteGu · 2026-04-01T02:48:12Z

/test ci/centos/mini-e2e-helm/k8s-1.34

YiteGu · 2026-04-01T05:55:51Z

helm test successful @nixpanic @iPraveenParihar

mergify bot added the component/rbd Issues related to RBD label Mar 6, 2026

YiteGu force-pushed the support-vac-feature-for-rbd branch 3 times, most recently from b78efe5 to 36c1c2f Compare March 6, 2026 11:43

YiteGu mentioned this pull request Mar 6, 2026

Use MutableParameters for NBD QOS in CreateVolume #5750

Closed

YiteGu force-pushed the support-vac-feature-for-rbd branch 4 times, most recently from c1e4f0a to dffa21e Compare March 9, 2026 13:41

nixpanic reviewed Mar 10, 2026

View reviewed changes

internal/rbd/controllerserver.go Outdated Show resolved Hide resolved

internal/rbd/controllerserver.go Show resolved Hide resolved

e2e/rbd.go Outdated Show resolved Hide resolved

internal/util/idlocker.go Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch 3 times, most recently from cb885ba to a8df95a Compare March 11, 2026 05:13

YiteGu force-pushed the support-vac-feature-for-rbd branch 2 times, most recently from 43fa44e to 1a4ade8 Compare March 11, 2026 06:25

YiteGu requested a review from nixpanic March 11, 2026 07:36

YiteGu force-pushed the support-vac-feature-for-rbd branch from f7b6685 to e786b28 Compare March 11, 2026 07:44

nixpanic reviewed Mar 11, 2026

View reviewed changes

internal/rbd/controllerserver.go Outdated Show resolved Hide resolved

examples/rbd/storageclass.yaml Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch from e786b28 to 650a107 Compare March 11, 2026 11:46

nixpanic reviewed Mar 11, 2026

View reviewed changes

PendingReleaseNotes.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

docs/volumeattributesclass.md Outdated Show resolved Hide resolved

YiteGu force-pushed the support-vac-feature-for-rbd branch 2 times, most recently from b2d68cb to a79ce62 Compare March 16, 2026 02:52

YiteGu requested a review from nixpanic March 16, 2026 03:34

nixpanic approved these changes Mar 16, 2026

View reviewed changes

nixpanic requested a review from a team March 16, 2026 09:05

nixpanic added the enhancement New feature or request label Mar 16, 2026

nixpanic requested changes Mar 16, 2026

View reviewed changes

internal/rbd/controllerserver.go Show resolved Hide resolved

ceph-csi-bot force-pushed the support-vac-feature-for-rbd branch from c0796db to fb16898 Compare March 31, 2026 04:43

iPraveenParihar added the ok-to-test Label to trigger E2E tests label Mar 31, 2026

ceph-csi-bot removed the ok-to-test Label to trigger E2E tests label Mar 31, 2026

YiteGu force-pushed the support-vac-feature-for-rbd branch from fb16898 to 0ec620e Compare April 1, 2026 02:40

deploy: add RBAC for csi-provisioner to access volumeattributesclass

bab1bd9

Signed-off-by: Yite Gu <guyite@bytedance.com>

YiteGu force-pushed the support-vac-feature-for-rbd branch from 0ec620e to bab1bd9 Compare April 1, 2026 02:44

Conversation

YiteGu commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe what this PR does

Is there anything that requires special attention

Related issues

Future concerns

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Mar 10, 2026

Uh oh!

mergify bot commented Mar 11, 2026

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

YiteGu commented Mar 11, 2026

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nixpanic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nixpanic commented Mar 30, 2026

Uh oh!

iPraveenParihar commented Mar 31, 2026

Uh oh!

mergify bot commented Mar 31, 2026

✅ Branch has been successfully rebased

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

ceph-csi-bot commented Mar 31, 2026

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

nixpanic commented Mar 31, 2026

Uh oh!

iPraveenParihar commented Mar 31, 2026

Uh oh!

nixpanic commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YiteGu commented Apr 1, 2026

Uh oh!

YiteGu commented Apr 1, 2026

Uh oh!

YiteGu commented Apr 1, 2026

Uh oh!

YiteGu commented Mar 6, 2026 •

edited

Loading

nixpanic commented Mar 31, 2026 •

edited

Loading