Disk size does not update in `kubectl get pvc` after resizing

[WizardBit]

After deploying Canonical k8s with ceph-csi I tried to resize a disk since it was getting full and noticed that the resize does occur, but the output from kubectl it is not reflecting this change.

Command used to resize storage disk

sudo k8s kubectl patch pvc prometheus-database-3fe23980-prometheus-0 \
    -n cos --type='merge' -p '{"spec": {"resources": {"requests": {"storage": "700Gi"}}}}'

persistentvolumeclaim/prometheus-database-3fe23980-prometheus-0 patched

Output of pvc does not reflect disk resize

sudo k8s kubectl get pvc -n cos
NAME                                                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
...
prometheus-database-3fe23980-prometheus-0           Bound    pvc-4b69b147-eb20-4df9-8bc5-23a1c77e7c2e   1Gi        RWO            ceph-xfs       <unset>                 27d

Container shows resize was successful

sudo k8s kubectl exec -n cos -c prometheus prometheus-0 -- df -h /var/lib/prometheus
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd0       700G  1.6G  699G   1% /var/lib/prometheus

It shows that k8s is waiting for the controller to respond back after the resize.

kubectl get events -n cos --sort-by=.metadata.creationTimestamp
LAST SEEN   TYPE     REASON                     OBJECT                                                            MESSAGE
5m49s       Normal   ExternalExpanding          persistentvolumeclaim/prometheus-configurations-24155b26-prometheus-0   waiting for an external controller to expand this PVC
3m10s       Normal   FileSystemResizeRequired   persistentvolumeclaim/prometheus-configurations-24155b26-prometheus-0   Require file system resize of volume on node
3m10s       Normal   Resizing                   persistentvolumeclaim/prometheus-configurations-24155b26-prometheus-0   External resizer is resizing volume pvc-4b69b147-eb20-4df9-8bc5-23a1c77e7c2e

Charm Versions

• ceph-csi 1.32/stable Rev 70
• ceph-mon squid/stable Rev 268
• ceph-osd squid/stable Rev 632
• k8s 1.32/stable Rev 986

[sed-i]

Related: juju/juju#19829

[addyess]

I think we need more information.

please check that allowVolumeExpansion: true on the StorageClass

i tested this to resize from 1Gi -> 2Gi and verified that it does happen.

Maybe check logs from the resizer controller?

[WizardBit]

I've confirmed that allowVolumeExpansion is set to true.

NAME                            PROVISIONER              RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
ceph-ext4                       rbd.csi.ceph.com         Delete          Immediate              true                   41d
ceph-xfs (default)              rbd.csi.ceph.com         Delete          Immediate              true                   41d
csi-rawfile-default (default)   rawfile.csi.openebs.io   Delete          WaitForFirstConsumer   true                   41d

  Type                      Status  LastProbeTime                     LastTransitionTime                Reason  Message
  ----                      ------  -----------------                 ------------------                ------  -------
  FileSystemResizePending   True    Mon, 01 Jan 0001 00:00:00 +0000   Fri, 29 Aug 2025 18:25:27 +0000           Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Events:
  Type    Reason                    Age                  From                               Message
  ----    ------                    ----                 ----                               -------
  Normal  Resizing                  89s (x4193 over 9d)  external-resizer rbd.csi.ceph.com  External resizer is resizing volume pvc-4b69b147-eb20-4df9-8bc5-23a1c77e7c2e
  Normal  FileSystemResizeRequired  86s (x4193 over 9d)  external-resizer rbd.csi.ceph.com  Require file system resize of volume on node

We see this message, but deleting the pod and having it re-created does not resolve the issue.

[WizardBit]

I took some inspiration from #49 and decided to try release 3.13.0 by setting juju config ceph-csi release=v3.13.0 and found that it did resolve this issue. I can easily reproduce this error from switching between v3.13.0 and v3.9.0, which seems to be the affected version.

Update: In a different environment changing the release version did not work as expected.