snapshot completes, volumesnapshotcontents hang not readytouse

[wmiller112]

/kind bug

What happened?
New volumesnapshotcontents periodically get stuck at ReadyToUse: false, significantly beyond (20+ minutes) being ready/complete in AWS. csi-snapshotter shows the appropriate logs for Creating snapshot for content, but eventually stop, leaving the volumesnapshotcontent not ready. Terminating the leader pod, triggering a new leader, immediately results in the volumesnapshotcontent transitioning to ready.

What you expected to happen?
The volumesnapshotcontents should become ready when the snapshot is ready/complete is aws without having to roll the pods

How to reproduce it (as minimally and precisely as possible)?

• Create a new volumesnapshot
• Get the resulting volumesnapshotcontents to retrieve the snapshotHandle
• Watch the volumesnapshotcontents
• observe the snapshot creation in aws until success
• Volumesnapshotcontent remains readytouse: false
• csi-snapshotter logs stop reporting Creating snapshot for content

Environment

• Kubernetes version (use kubectl version):

    Client Version: v1.32.3
    Kustomize Version: v5.5.0
    Server Version: v1.32.8-eks-e386d34

• Driver version:
  • chart version 2.49.1
  • driver image 1.49.1

[mdzraf]

Hello @wmiller112 , thank you for submitting this issue. I attempted to reproduce the issue with the steps you suggested above but I was not able to. When I tried, it actually succeeded snapshot creation and Ready to Use was set to true.

Status:
...
  Ready To Use:                        true
  Restore Size:                        4Gi
Events:
  Type    Reason            Age   From                 Message
  ----    ------            ----  ----                 -------
  Normal  CreatingSnapshot  76s   snapshot-controller  Waiting for a snapshot default/ebs-volume-snapshot to be created by the CSI driver.
  Normal  SnapshotCreated   75s   snapshot-controller  Snapshot default/ebs-volume-snapshot was successfully created by the CSI driver.
  Normal  SnapshotReady     8s    snapshot-controller  Snapshot default/ebs-volume-snapshot is ready to use.

Are you consistently able to reproduce the issue? If so is there any other information about your configuration or snapshot(s) that can help us to better reproduce the issue?