Cross-pool snapshot restore fallback when source pool has insufficient space

[yugchaudhari]

Is your feature request related to a problem? Please describe.

When creating a volume from a snapshot (CSI CreateVolume with snapshot source), Mayastor requires the new volume to be placed on the same pool as the snapshot's replica (because CoW snapshots are blobstore-local). If that pool is full, the operation fails with 507 Insufficient Storage, even when other pools in the cluster have terabytes of free space.

In our production KubeVirt environment, we use golden image snapshots to clone VM disks. The golden image's replicas were on a pool with 3.5 TiB used / 3.5 TiB capacity. Every VM provisioning attempt failed indefinitely, while other pools on the same and different nodes had 2-7 TiB free. The CSI driver kept retrying the same doomed restore for 38+ minutes (144 retry events) until manual intervention.

Describe the solution you'd like

Add fallback logic in the CreateVolume handler when restoring from a snapshot:

1. Identify snapshot replica pools
2. For each pool, check available capacity vs requested volume size
3. If sufficient space → proceed with local CoW restore (existing fast path, instant)
4. If insufficient space on ALL snapshot replica pools → fall back to "full copy restore":
   • Allocate the new volume on a different pool that matches the volume's topology constraints and has sufficient capacity
   • Copy data from the snapshot replica to the new volume's replica over the network
   • Complete the CreateVolume call successfully

This is architecturally feasible because Mayastor already has the building blocks:

• Cross-node/cross-pool data copy: used during replica rebuilds (self-heal)
• Network-based replication: replicas are routinely copied between pools on different nodes
• HotSpareReconciler infrastructure: detects a missing replica and copies data from a healthy replica to a new one on a different pool

What's missing is the logic to use that same mechanism during snapshot restore. The CreateVolume from snapshot handler currently has a hardcoded assumption: "restore must happen on the same pool as the snapshot."

Describe alternatives you've considered

1. Return a non-retriable error code: If cross-pool restore is too complex, return FailedPrecondition instead of ResourceExhausted so the external-provisioner stops retrying and the orchestrator (CDI) can take alternative action. Currently ResourceExhausted is treated as retriable, causing infinite retry loops.

2. CDI-level fallback: I've filed a comment on kubevirt/containerized-data-importer#4068 requesting CDI to fall back to host-assisted copy when snapshot clone fails. This works but is slower (involves a copy pod) and doesn't leverage Mayastor's native replication.

3. Operational workaround: Ensure source image pools always have free space. This is what we do today, but it's fragile and requires constant capacity monitoring.

Additional context

Approach	Speed	Space requirement
Same-pool CoW restore (current)	Instant (metadata only)	Same pool must have space
Cross-pool full copy (proposed fallback)	Slower (network copy)	Any pool with space works

The fallback is slower but succeeds, which is better than indefinite failure.

Environment:

• Mayastor: v2.10.0
• Kubernetes: v1.34.3
• CDI: v1.64.0
• Pool configuration: 14 pools across 6 nodes (3.5 TiB and 7 TiB pools)

Related: #1895 (snapshot rebuild when pool is offline, similar problem, different trigger)

[yugchaudhari]

@tiagolobocastro @Abhinandan-Purkait

I'd like to propose an implementation approach for this and offer to take it up if you're aligned.

Proposed Approach: SnapshotRebuildJob-based fallback in CreateVolume

After investigating the codebase, I believe we can implement this with zero data-plane changes by reusing the existing SnapshotRebuildJob mechanism:

Modified CreateVolume flow:

CreateVolume(content_source=snapshot):

1. Try same-pool CoW clone (existing fast path — vbdev_lvol_create_clone_ext)
   → If succeeds: done (instantaneous, no change to current behavior)

2. If pool full (ResourceExhausted):
   a. Check if a SnapshotRebuildJob already exists for this volume
      - If completed → create nexus, return success
      - If still running → return gRPC Aborted ("operation in progress, retry")
   b. If no existing rebuild:
      - Create an empty thin replica on a pool with space (standard create_replica)
      - Start SnapshotRebuildJob(source=snapshot_bdev_uri, target=new_replica_uri)
      - Return gRPC Aborted
   c. External-provisioner retries with backoff → hits (2a) → eventually completes

Why this works:

• SnapshotRebuildJob already reads from snapshot bdevs and writes to replicas on different pools, this is
  what it's designed for
• No data-plane changes, just control-plane orchestration in clone_operations.rs
• CSI-compliant, Aborted means "operation in progress, retry later" per gRPC/CSI semantics
• Crash-safe, SnapshotRebuildJob persists in io-engine, empty replica tracked in etcd; on restart, next CreateVolume retry picks up where it left off
• Performance, for our 34 GiB golden image: ~30-40 seconds total (vs instant for same-pool, but succeeds vs infinite failure)

Scope:

• ~200-400 lines in control-plane (clone_operations.rs + pool scheduling for cross-pool case)
• Similar patterns to the offline volume rebuild work I'm already doing (openebs/openebs#4208)

Questions for you:

1. Does this approach align with how you'd want this to work? Any concerns with using Aborted for the retry semantic?
2. Is there anything in the SnapshotRebuildJob lifecycle that wouldn't work when triggered from a CreateVolume context (vs the existing snapshot-rebuild use cases)?
3. Should I go ahead and draft an OEP for this?

I'm already getting familiar with the volume lifecycle and control-plane patterns through the offline rebuild work. Happy to take this up if it fits the roadmap.

[tiagolobocastro]

I'm travelling for another week still, so don't have a lot of time to review these but some initial questions.

Are these golden images snapshots 1-r or multi-replica?
Bear in mind I have no idea on how these golden images are created so please excuse me if I'm totally off, but if you could have a multi-replica snapshot (with more replicas than you need), then it should be possible to create a restore with the the number of replicas you need, we'd just pick the snapshots from the pools with sufficient space.

Also IIRC we already support creating restores with replica count larger than snapshot count, or is it only with less? (someone needs to confirm here).

Also it's worth considering if you ever even want to use COW if you actually need a full copy for your use case (which could be automated to be lazy I guess)

Otherwise I think the proposed changes would work!

Currently ResourceExhausted is treated as retriable, causing infinite retry loops.

Yes sadly this is the case. There was some PR to fix but it never got merged: kubernetes-csi/external-snapshotter#1334
If you could also try to revive that it would be awesome.
Otherwise, maybe we could reconsider swapping the error code..

[yugchaudhari]

Thanks for the response even while traveling, @tiagolobocastro, appreciate it!

Golden image setup

Our golden images are 2-replica (num_replicas: 2). Snapshots inherit this, each snapshot has 2 replica snapshots across 2 pools.

Pool selection during restore, code investigation

I dug into clone_operations.rs and I think your first question hits the exact failure mode. Here's what I found:

The create() method in SnapshotCloneOp iterates over candidates and does handle per-replica failures gracefully (logs error, continues to next). So if one pool is full, it tries the next, good.

But the check in setup() is the bottleneck:

if volume.num_replicas > clonable_snapshots.len() as u8 {
    return Err(SvcError::InsufficientSnapshotsForClone { ... });
}

And later:

if pools.len() < new_volume.num_replicas as usize || pools.is_empty() {
    return Err(SvcError::NoSnapshotPools { ... });
}

For a 2-replica restore from a 2-replica snapshot: if the CloneVolumeSnapshot scheduler filters out the full pool (capacity check), you're left with 1 eligible pool but need 2 → NoSnapshotPools error. If it doesn't filter by capacity, both pools pass scheduling, the first clone fails at io-engine level (ResourceExhausted), but the second succeeds, leaving you with only 1 replica created vs 2 needed.

Either way, the result is a failed CreateVolume.

There's also this TODO in the code that confirms the gap:

// todo: need to add new replica and do full rebuild, if clonable snapshots
// count is less than volume replicas count.

Your suggestion: create fewer clones + rebuild the rest

This is exactly what's needed. For our scenario:

• Snapshot has 2 replicas (pool A = full, pool B = has space)
• Create 1 CoW clone from pool B's snapshot replica
• Let self-heal rebuild the 2nd replica on any pool with space (normal replica rebuild, no snapshot involvement)

This would be a relatively small change:

1. In setup() relax the check: allow clonable_snapshots.len() < num_replicas (as long as ≥ 1)
2. In create() create as many CoW clones as possible from available snapshot replicas
3. After volume creation, the existing HotSpareReconciler detects the volume has fewer replicas than num_replicas and triggers a normal rebuild to bring it up to count

The volume would be temporarily degraded (1/2 replicas) until rebuild completes, but it would be usable immediately, same as any newly-created volume during its first rebuild.

Do we ever want COW vs full copy?

For our use case (golden images cloned 20-50 times, VMs diverge fully within hours), we don't benefit much from long-term COW sharing. However, the instant provisioning speed of COW is critical, we need VMs to boot in seconds, not wait 30-40s for a full copy.

So ideally: COW clone for instant availability + optional background full-copy to eventually eliminate snapshot dependency. But that's a separate enhancement, for now, COW clone with relaxed replica count (+ self-heal rebuild) would fully solve our problem.

External-snapshotter PR

I'll look into reviving kubernetes-csi/external-snapshotter#1334. Making ResourceExhausted non-retriable (or distinguishable) would help the broader ecosystem. In the meantime, swapping to FailedPrecondition on our side could be a quick win as well.

Summary / next steps

Based on your feedback, I think the minimal fix is:

1. Relax the replica count requirement in clone_operations.rs, allow creating a restore with fewer CoW clones than num_replicas, as long as at least 1 snapshot replica pool has space
2. Let self-heal handle the rest, HotSpareReconciler rebuilds the missing replicas via normal rebuild

This is much simpler than the SnapshotRebuildJob approach I originally proposed (that remains the fallback for cases where ALL snapshot replica pools are full).

Should I draft an OEP for this approach, or is this small enough to go straight to a PR?