Backup job failed for Google S3 Remote #10264

Are you using XOA or XO from the sources?

XO from the sources

Which release channel?

latest

Provide your commit number

d53d2

Describe the bug

On an S3 remote backed by Google Cloud Storage, cleanVm can detect orphaned VHD directories but can never delete them. Every backup run logs the same orphans, the job still reports success, and the orphaned data accumulates indefinitely. In my case 50 orphaned VHD directories built up over ~7 weeks under a single VDI before I noticed.

This is not a permissions problem. The service account behind the HMAC key holds both Storage Admin and Storage Object Admin; the bucket has no retention policy, no object holds, and versioning is off.

Root cause

RemoteDiskLineage.#cleanOrphanDataFiles deletes orphans with an unlink-then-fallback pattern (@xen-orchestra/backup-archive/src/RemoteDiskLineage.mts):

await this.#handler.unlink(item).catch(async err => {
  if (err?.code === 'EISDIR') {
    await this.#handler.rmtree(item)      // the call that actually deletes it
  } else {
    this.#opts.logWarn('failed to delete orphaned data file', { path: item, error: err })
  }
})

Because S3Handler.useVhdDirectory() returns true, item is always a key prefix (containing header, footer, bat, blocks/…), never a single object. So reaching rmtree depends entirely on _unlink throwing EISDIR (@xen-orchestra/fs/src/s3.js):

async _unlink(path) {
  await this.#s3.send(new DeleteObjectCommand(this.#createParams(path)))

  if (await this.#isNotEmptyDir(path)) {
    const error = new Error(`EISDIR: illegal operation on a directory, unlink '${path}'`)
    error.code = 'EISDIR'
    ...

On AWS S3, deleting a key that doesn't exist returns 204, so execution falls through to the #isNotEmptyDir check, EISDIR is thrown, and rmtree runs.

Google Cloud Storage instead returns 404 NoSuchKey for a DELETE on a nonexistent object — documented behaviour, not a transient fault. The SDK throws, _unlink propagates it, ignoreEnoent in abstract.js doesn't match (the code is NoSuchKey, not ENOENT), the EISDIR branch is never reached, and rmtree is never called.

The result is that the deletion path is unreachable on GCS by construction, regardless of IAM configuration.

Error message

{
  "data": {
    "path": "/xo-vm-backups/<vm-uuid>/vdis/<job-uuid>/<vdi-uuid>/data/20260330T183021Z.vhd",
    "error": {
      "$fault": "client",
      "$metadata": {
        "httpStatusCode": 404,
        "attempts": 1,
        "totalRetryDelay": 0
      },
      "name": "NoSuchKey",
      "message": "The specified key does not exist.",
      "Code": "NoSuchKey",
      "Details": "No such object: <bucket>/xo-vm-backups/<vm-uuid>/vdis/<job-uuid>/<vdi-uuid>/data/20260330T183021Z.vhd"
    }
  },
  "message": "failed to delete orphaned data file"
}

One backup log contained 100 warnings — 50 orphaned data file / 50 failed to delete orphaned data file, each cleanup pass running twice — and every single failure was NoSuchKey / 404. No AccessDenied, no other error code.

Error message

undefined

To reproduce

1. Create a GCS bucket and an HMAC key for a service account with Storage Object Admin.
2. In XO, add an S3 remote pointing at storage.googleapis.com with that bucket.
3. Run an incremental (delta) backup job against it for long enough that retention orphans at least one VHD directory.
4. Watch the Clean VM directory task in the backup log.
5. Every run logs orphaned data file immediately followed by failed to delete orphaned data file with a 404 NoSuchKey, and the orphans are never removed.

Expected behavior

_unlink should treat "no object at this key" as a non-error, matching AWS's 204 semantics, so that the #isNotEmptyDir check runs and callers get EISDIR for VHD directories. Orphaned VHD directories should then be removed by rmtree as they are on AWS.

Suggested fix

Swallowing the 404 inside _unlink — rather than mapping it to ENOENT — is what preserves the fallback. Mapping to ENOENT would let ignoreEnoent silently absorb it and rmtree would still never run, turning a loud failure into a silent one.

Diff

--- a/@xen-orchestra/fs/src/s3.js
+++ b/@xen-orchestra/fs/src/s3.js
@@ -296,7 +296,17 @@
   }
 
   async _unlink(path) {
-    await this.#s3.send(new DeleteObjectCommand(this.#createParams(path)))
+    try {
+      await this.#s3.send(new DeleteObjectCommand(this.#createParams(path)))
+    } catch (error) {
+      // AWS S3 answers 204 when deleting a key that does not exist, but some providers
+      // (e.g. Google Cloud Storage) answer 404 NoSuchKey instead. Both mean "there is no
+      // object at this key", which is not an error here: the path may be a VHD directory,
+      // which is detected by the check below and deleted by the caller via rmtree().
+      if (error.name !== 'NoSuchKey' && error.$metadata?.httpStatusCode !== 404) {
+        throw error
+      }
+    }
 
     if (await this.#isNotEmptyDir(path)) {
       const error = new Error(`EISDIR: illegal operation on a directory, unlink '${path}'`)

I've been running this patch locally and orphan cleanup now completes against GCS. Happy to open it as a PR if you'd like.

Screenshots

No response

Node

v22.22.2

Hypervisor

XCP-ng 8.2.0

Additional context

Two related observations, both separate from the bug above:

The job reports success. Cleanup failures are logged only as warnings, so a remote can silently accumulate undeletable data forever without anything surfacing. Worth considering whether repeated cleanup failures should escalate.
_rmtree ignores per-object errors. DeleteObjectsCommand's response is not inspected, so entries in the Errors array are treated as successes. GCS's multi-object delete is explicitly non-atomic and can partially fail, so partial failures there would be invisible. Not what caused my issue — _unlink fails long before rmtree — but it's the next failure mode in the same code path once this fix lands.

For reference, the same GCS S3-compatibility gap has bitten other backup tools: Proxmox Backup Server and Nextcloud, though in those cases it was the multi-object delete endpoint rather than single-object DELETE.