All backups are stuck in Finalizing

[andrea-maruccia-n]

This is my Velero version:

>velero version
Client:
        Version: v1.18.0
        Git commit: -
Server:
        Version: v1.18.0

All the backups are stuck in Finalizing

>velero backup get
NAME                                   STATUS                      ERRORS   WARNINGS   CREATED                          EXPIRES   STORAGE LOCATION   QUEUE POSITION   SELECTOR
manual-mon-ama-2026-04-21-9-53         Finalizing                  0        0          2026-04-21 09:53:11 +0200 CEST   2h        aws                                 <none>
velero-cluster-backup-20260421060026   Finalizing                  0        0          2026-04-21 08:00:26 +0200 CEST   11h       aws                                 <none>
velero-cluster-backup-20260421030026   Finalizing                  0        0          2026-04-21 05:00:26 +0200 CEST   8h        aws                                 <none>
velero-cluster-backup-20260421000026   FinalizingPartiallyFailed   6        0          2026-04-21 02:00:26 +0200 CEST   5h        aws                                 <none>
velero-cluster-backup-20260420210025   Finalizing                  0        0          2026-04-20 23:00:25 +0200 CEST   2h        aws                                 <none>
velero-cluster-backup-20260420191225   FinalizingPartiallyFailed   4        0          2026-04-20 21:12:26 +0200 CEST   35m       aws                                 <none>

when I check the details:

>velero backup describe manual-mon-ama-2026-04-21-9-53 --details
Name:         mmanual-mon-ama-2026-04-21-9-53
Namespace:    backup
Labels:       velero.io/storage-location=aws
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.35.2-eks-f69f56f
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=35

Phase:  Finalizing


Namespaces:
  Included:  monitoring
  Excluded:  <none>

Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io

Label selector:  <none>

Or label selector:  <none>

Storage Location:  aws

Velero-Native Snapshot PVs:    auto
File System Backup (Default):  false
Snapshot Move Data:            true
Data Mover:                    velero

TTL:  3h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2026-04-21 09:53:11 +0200 CEST
Completed:  <n/a>

Expiration:  2026-04-21 12:53:11 +0200 CEST

Total items to be backed up:  383
Items backed up:              383

Backup Item Operations:
  Operation for persistentvolumeclaims monitoring/prometheus-grafana:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               <snipped>
    Items to Update:
                           datauploads.velero.io backup/manual-mon-ama-2026-04-21-9-53-rp4ss
    Phase:                 Completed
    Progress:              190316422 of 190316422 complete (Bytes)
    Progress description:  Completed
    Created:               2026-04-21 09:53:19 +0200 CEST
    Started:               2026-04-21 09:54:38 +0200 CEST
    Updated:               2026-04-21 09:55:01 +0200 CEST
  Operation for persistentvolumeclaims monitoring/prometheus-monitoring-prometheus-db-prometheus-monitoring-prometheus-0:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               <snipped>
    Items to Update:
                           datauploads.velero.io backup/manual-mon-ama-2026-04-21-9-53-48gtf
    Phase:                 Completed
    Progress:              26005812150 of 26005812150 complete (Bytes)
    Progress description:  Completed
    Created:               2026-04-21 09:53:24 +0200 CEST
    Started:               2026-04-21 09:55:50 +0200 CEST
    Updated:               2026-04-21 09:58:57 +0200 CEST
...
<snipped>
...
Resource List:
...
<snipped>
...

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    monitoring/prometheus-grafana:
      Data Movement:
        Operation ID: du-b438e240-79eb-4b63-b7a4-c8b982b6579a.52a8e8f8-ea3c-4803a2264
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: 
    monitoring/prometheus-monitoring-prometheus-db-prometheus-monitoring-prometheus-0:
      Data Movement:
        Operation ID: du-b438e240-79eb-4b63-b7a4-c8b982b6579a.7a5f2af8-34b0-468057a67
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 0
        Result: 
    ...
    <snipped>
    ...

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

As you can see I cannot find any useful information on why it's stuck or remains in the Finalizing status.
I also check logs but I couldn't find anything "meaningful" with my knowledge.

Does anybody know how to further debug this? what should I be looking at?

[andrea-maruccia-n]

I have debugged this further:

• I have checked all DataUploads status and they were all completed
• I have checked the logs for errors:

I found these 3 errors at the end of the logs

time="2026-04-22T06:08:31Z" level=error msg="error encountered while scanning stdout" cmd=/velero error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:90"
time="2026-04-22T06:57:57Z" level=error msg="error encountered while scanning stdout" backupLocation=backup/aws cmd=/plugins/velero-plugin-for-aws controller=backup-sync error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:90"
time="2026-04-22T07:13:01Z" level=error msg="error encountered while scanning stdout" backup-storage-location=backup/aws cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location error="read |0: file already closed" logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:90"

According to claude this means the subprocess died

Root cause found. The velero-plugin-for-aws plugin process is crashing (file already closed = the plugin subprocess died and its stdout pipe was closed). This happens during the final step where Velero needs the AWS plugin to write backup metadata to S3 — so it never completes.

At this point I restarted the Velero pod to give it a chance to reconcile and it succeeded.

Next step is to understand why the process crashed, maybe related to memory limits on the pod(s)
Edit: I removed the limits and it worked without crashing. Probably this is the fix!