Volume expansion does not increase max inodes

[tosseth]

Describe the bug

• Running Scale CSI 2.5.0
• Using an Openshift Scale CSI storage class with independent filesets.
• I increased the size of a PVC from 1Gi to 20Gi.
• The size increased properly; however, the max inodes did not increase for the fileset. It should have increased from 100096 to 200192.

How to Reproduce?

• Create a small PVC volume (i.e. 1Gi) with a Scale storage class using independent filesets.
• Increase the size of the PVC beyond the 10Gi CSI boundary for increasing max inodes (i.e. 20Gi)

Expected behavior

The max inodes for the fileset should have increased from 100096 to 200192.

Data Collection and Debugging

Storage class creation details (with "allowVolumeExpansion"):

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ibm-spectrum-scale-volexp-sc
provisioner: spectrumscale.csi.ibm.com
parameters:
  clusterId: "13261356876944010532"
  volBackendFs: esccfs
  permissions: "777"
reclaimPolicy: Delete
allowVolumeExpansion: true

Create 1Gi PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ibm-spectrum-scale-test-vol-to-expand
spec:
  storageClassName: ibm-spectrum-scale-volexp-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

PVC created and bound:

$ oc get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
ibm-spectrum-scale-test-vol-to-expand   Bound    pvc-fad749fc-1b44-4f9d-b5d5-435045790b23   1Gi        RWX            ibm-spectrum-scale-volexp-sc     8m37s

PV created:

$ oc get pv |grep -E "ibm-spectrum-scale-test-vol-to-expand|NAME"
NAME                                              CAPACITY      ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                                                                                        STORAGECLASS                     REASON   AGE
pvc-fad749fc-1b44-4f9d-b5d5-435045790b23          1Gi           RWX            Delete           Bound         astewarproject/ibm-spectrum-scale-test-vol-to-expand                                         ibm-spectrum-scale-volexp-sc              21s

On Scale storage side, look at the fileset details, for quota and inodes:

sh-4.4# mmlsquota -j pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 esccfs
                         Block Limits                                    |     File Limits
Filesystem type             KB      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
esccfs     FILESET           0    1048576    1048576          0     none |        2       0        0        0     none
sh-4.4#

sh-4.4# mmlsfileset esccfs pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 -i
Collecting fileset usage information ...
Filesets in file system 'esccfs':
Name                     Status    Path                                         InodeSpace      MaxInodes    AllocInodes     UsedInodes
pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 Linked /mnt/esccfs/pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 450       100096 9216            2

Now increase the PVC size from 1Gi to 20Gi:

oc edit pvc ibm-spectrum-scale-test-vol-to-expand:

spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 20Gi <----

From Openshift, check the PVC/PV size. It increased to 20Gi:

$ oc get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     AGE
ibm-spectrum-scale-test-vol-to-expand   Bound    pvc-fad749fc-1b44-4f9d-b5d5-435045790b23   20Gi       RWX            ibm-spectrum-scale-volexp-sc     10m

$ oc get pv |grep ibm-spectrum-scale-test-vol-to-expand
pvc-fad749fc-1b44-4f9d-b5d5-435045790b23          20Gi          RWX            Delete           Bound         astewarproject/ibm-spectrum-scale-test-vol-to-expand                                         ibm-spectrum-scale-volexp-sc              10m

Describe of PVC shows the events for the resize also:

oc describe pvc ibm-spectrum-scale-test-vol-to-expand
..
Events:
  Type     Reason                  Age                From                                                                                                 Message
  ----     ------                  ----               ----                                                                                                 -------
  Normal   Provisioning            11m                spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-0_713116d4-6274-462d-906d-cd3ca228c32e  External provisioner is provisioning volume for claim "astewarproject/ibm-spectrum-scale-test-vol-to-expand"
  Normal   ExternalProvisioning    11m (x3 over 11m)  persistentvolume-controller                                                                          waiting for a volume to be created, either by external provisioner "spectrumscale.csi.ibm.com" or manually created by system administrator
  Normal   ProvisioningSucceeded   11m                spectrumscale.csi.ibm.com_ibm-spectrum-scale-csi-provisioner-0_713116d4-6274-462d-906d-cd3ca228c32e  Successfully provisioned volume pvc-fad749fc-1b44-4f9d-b5d5-435045790b23
  Warning  ExternalExpanding       77s                volume_expand                                                                                        Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing                77s                external-resizer spectrumscale.csi.ibm.com                                                           External resizer is resizing volume pvc-fad749fc-1b44-4f9d-b5d5-435045790b23
  Normal   VolumeResizeSuccessful  75s                external-resizer spectrumscale.csi.ibm.com                                                           Resize volume succeeded

Now, after waiting a few minutes, check the fileset details on the storage cluster again:

sh-4.4# mmlsquota -j pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 esccfs
                         Block Limits                                    |     File Limits
Filesystem type             KB      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
esccfs     FILESET           0   20971520   20971520          0     none |        2       0        0        0     none
sh-4.4#

sh-4.4# mmlsfileset esccfs pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 -i
Collecting fileset usage information ...
Filesets in file system 'esccfs':
Name                     Status    Path                                         InodeSpace      MaxInodes    AllocInodes     UsedInodes
pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 Linked /mnt/esccfs/pvc-fad749fc-1b44-4f9d-b5d5-435045790b23 450       100096 9216            2
sh-4.4#

Scale CSI resizer pod logs:

oc logs ibm-spectrum-scale-csi-resizer-0 -n ibm-spectrum-scale-csi --since=20m

I0426 15:21:43.322663       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 32 items received
I0426 15:24:03.317058       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolume total 8 items received
I0426 15:25:53.711326       1 controller.go:291] Started PVC processing "astewarproject/ibm-spectrum-scale-test-vol-to-expand"
I0426 15:25:53.718393       1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0426 15:25:53.718403       1 connection.go:184] GRPC request: {}
I0426 15:25:53.718535       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"astewarproject", Name:"ibm-spectrum-scale-test-vol-to-expand", UID:"fad749fc-1b44-4f9d-b5d5-435045790b23", APIVersion:"v1", ResourceVersion:"422822580", FieldPath:""}): type: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-fad749fc-1b44-4f9d-b5d5-435045790b23
I0426 15:25:53.719045       1 connection.go:186] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":7}}}]}
I0426 15:25:53.719168       1 connection.go:187] GRPC error: <nil>
I0426 15:25:53.719194       1 connection.go:183] GRPC call: /csi.v1.Controller/ControllerExpandVolume
I0426 15:25:53.719198       1 connection.go:184] GRPC request: {"capacity_range":{"required_bytes":21474836480},"volume_capability":{"AccessType":{"Mount":{"fs_type":"gpfs"}},"access_mode":{"mode":5}},"volume_id":"0;2;13261356876944010532;0A83001C:61EE70F2;;pvc-fad749fc-1b44-4f9d-b5d5-435045790b23;/mnt/esccfs/primary-fileset-esccfs-13261356876944010532/.volumes/pvc-fad749fc-1b44-4f9d-b5d5-435045790b23"}
I0426 15:25:55.770647       1 connection.go:186] GRPC response: {"capacity_bytes":21474836480}
I0426 15:25:55.770719       1 connection.go:187] GRPC error: <nil>
I0426 15:25:55.770729       1 controller.go:468] Resize volume succeeded for volume "pvc-fad749fc-1b44-4f9d-b5d5-435045790b23", start to update PV's capacity
I0426 15:25:55.770737       1 controller.go:570] Resize volume succeeded for volume "pvc-fad749fc-1b44-4f9d-b5d5-435045790b23", start to update PV's capacity
I0426 15:25:55.788317       1 controller.go:474] Update capacity of PV "pvc-fad749fc-1b44-4f9d-b5d5-435045790b23" to 20Gi succeeded
I0426 15:25:55.792488       1 controller.go:533] Resize PVC "astewarproject/ibm-spectrum-scale-test-vol-to-expand" finished
I0426 15:25:55.792539       1 controller.go:291] Started PVC processing "astewarproject/ibm-spectrum-scale-test-vol-to-expand"
I0426 15:25:55.792552       1 controller.go:334] No need to resize PVC "astewarproject/ibm-spectrum-scale-test-vol-to-expand"
I0426 15:25:55.792551       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"astewarproject", Name:"ibm-spectrum-scale-test-vol-to-expand", UID:"fad749fc-1b44-4f9d-b5d5-435045790b23", APIVersion:"v1", ResourceVersion:"422822580", FieldPath:""}): type: 'Normal' reason: 'VolumeResizeSuccessful' Resize volume succeeded
I0426 15:30:39.325261       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolumeClaim total 45 items received
I0426 15:30:51.319919       1 reflector.go:535] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PersistentVolume total 9 items received

Environment Details

Openshift Scale details:

Spectrum Scale:
- Spectrum Scale CNSA version: 5.1.3.0
- Spectrum Scale CSI version: 2.5.0
- Spectrum Scale file system is local storage
- Spectrum Scale fs name: esccfs
- Spectrum Scale fs capacity: 50T [Used: 4.2T (9%) | Free: 46T]

Storage cluster level (Note: this is on Spectrum Fusion HCI)

sh-4.4# mmdiag --version

=== mmdiag: version ===
Current GPFS build: "5.1.3.0 ".
Built on Feb 17 2022 at 11:47:04
Running 19 days 18 hours 9 minutes 54 secs, pid 1432
sh-4.4#

Add labels

• Component:
• Severity:
• Customer Impact:
• Customer Probability:
• Phase:

Note : See labels for the labels

[tosseth]

Slack reference for indication that max inodes should increase in this scenario:

https://ibm-systems-storage.slack.com/archives/CGJTSKQ4U/p1648529623250609?thread_ts=1648502747.784779&cid=CGJTSKQ4U

[deeghuge]

Hi @tosseth , please share the CSI snap logs for this issue

[tosseth]

Hi @deeghuge
I recreated the problem on the same cluster (Spectrum Fusion HCI) and collected the CSI snap logs:
/u/DUMPS/scale-core/csi-705

Another interesting piece of data is I tried it on a cluster with CSNA/CSI with remote mount and it was successful, the max inodes increased as expected.

Then I tried the scenario on another Spectrum Fusion HCI cluster but it failed the same way as this cluster.

Perhaps there is an issue with Spectrum Fusion HCI, or maybe locally mounted storage (not remote storage)?