VolumeSnapshot is not getting deleted in false state, but gets deleted if valid VolumeSnapshot is created in same namespace.

[monimeht]

Describe the bug

VolumeSnapshot is not getting deleted in false state, but gets deleted if valid VolumeSnapshot is created in same namespace.

How to Reproduce?

1. Create sc version =2
2. create pvc.
3. create volumesnapshotclass , with incorrect snapwindow value like "one"
4. create volumesnapshot using above. (volumesnapshot created with false state)
5. delete volumesnapshot. (not getting deleted)
6. Create another volumesnapshot with valid parameters in volumesnapsot class.
7. Delete the above volumesnapshot .
8. Observe that the volumesnapshot with false state got deleted.

Expected behavior

VolumeSnapshot should get deleted.

Data Collection and Debugging

find attached the steps executed on cluster.
Bug_vol_snap_delete.txt

Builds used:
quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator:v2.5.0-100322
quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.5.0-100322

[monimeht]

Storage Class yaml:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sctest
provisioner: spectrumscale.csi.ibm.com
parameters:
volBackendFs: "fs1"
version: "2"
compression: "true"
reclaimPolicy: Delete
allowVolumeExpansion: false

PVC.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pvc1
spec:
accessModes:

• ReadWriteMany
  resources:
  requests:
  storage: 10Gi
  storageClassName: sctest

VolumesnapshotClass.yaml:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: snapclasstest
driver: spectrumscale.csi.ibm.com
parameters:
snapWindow: "one"
deletionPolicy: Delete

Volumesnapshot.yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: snapshot5
namespace: test
spec:
volumeSnapshotClassName: snapclasstest
source:
persistentVolumeClaimName: pvc1

[madhuthorat]

This is reproducible, looks to be snapshotter issue probably. The DeleteSnapshot() doesn't get called so obviously CSI Driver won't delete it.

Saw following error:
Warning SnapshotContentCheckandUpdateFailed 82s csi-snapshotter spectrumscale.csi.ibm.com Failed to check and update snapshot content: failed to remove VolumeSnapshotBeingCreated annotation from the content snapcontent-2072688c-9322-47c9-8128-0df262a9c672: snapshot controller failed to update snapcontent-2072688c-9322-47c9-8128-0df262a9c672 on API server: Operation cannot be fulfilled on volumesnapshotcontents.snapshot.storage.k8s.io "snapcontent-2072688c-9322-47c9-8128-0df262a9c672": the object has been modified; please apply your changes to the latest version and try again

[Jainbrt]

@monimeht can you please raise this issues in snapshotter community ?