Volume resize reconciler retries only nexus resize, never re-attempts failed replica resize

[cmrmahesh]

Describe the bug
We hit this in production — a volume resize from 150 GiB to 250 GiB got stuck permanently because one of the three replicas failed to expand (likely a transient gRPC issue), but the other two succeeded. After that, the reconciler keeps retrying resize_nexus every 5 minutes, which always fails because the one undersized replica is never re-expanded. The volume sat in this broken state for 7+ hours until we manually intervened.

The core issue: fixup_nexus_size in the nexus reconciler only retries the nexus resize — it never checks whether all child replicas are actually at the required size before doing so.

To Reproduce

1. Have a 3-replica volume
2. Trigger a volume resize (e.g., 150 GiB → 250 GiB)
3. Induce a failure on one replica's resize (transient gRPC timeout, io-engine briefly unreachable, etc.) while the other two succeed
4. Volume spec gets committed at 250 GiB
5. Reconciler picks up the nexus size mismatch and retries resize_nexus — fails with:
   Child nvmf://10.x.x.x:8420/nqn.2019-05.io.openebs: of nexus is too small: size = 314572800 x 512, required = 524288000 x 512
6. This repeats every 5 minutes forever. No resize_replica is ever issued for the stuck replica.

Expected behavior
The reconciler should notice the undersized replica and expand it before retrying the nexus resize. The volume should self-heal.

** OS info (please complete the following information):**

• Ubuntu 22.04
• Mayastor 2.10.0
• 6-node io-engine cluster, 3-replica volumes
• Kubernetes 1.28+

Additional context
One thing that made debugging harder: there's no REST endpoint to manually resize a replica (/v0/replicas/{id}/resize returns 404). The only workaround we found was deleting the replica and letting the rebuild happen at the correct size. Exposing a replica resize endpoint would be a nice operational escape hatch too.

[yugchaudhari]

Additional findings from investigation

We dug deeper into the io-engine and agent-core logs and found the root cause is more specific:

The control plane never sent a resize gRPC to the SnapshotClone replica.

Evidence:

• io-engine logs on cmp-bom-2 (hosting the stuck replica d68acc2c): zero resize requests received in the last 24 hours
• io-engine logs on cmp-bom-3 (hosting d8d9861d, Regular kind): Resized Lvol ... [thin 250.00 GiB] successfully logged every 5 minutes for 7+ hours
• At time of deletion, the stuck replica still showed [thin 150.00 GiB], never resized

Trigger condition:

• Volume was created from a snapshot (both replicas initially kind: SnapshotClone)
• On May 19, the reconciler faulted one replica (state_reason: AdminCommandFailed), scaled to 1, then scaled back to 2, creating a new Regular replica via rebuild
• The original SnapshotClone replica remained in the volume topology
• On May 25, resize was triggered, only the Regular replica received the resize gRPC

What we also tested:

• Created a fresh volume from snapshot with mixed topology (SnapshotClone + Regular after scale down/up): resize worked fine
• This means the bug is triggered by a specific internal state of the SnapshotClone replica after the AdminCommandFailed fault cycle, not just by mixed topology alone

Key question: Why does resize_volume_replicas() skip the SnapshotClone replica in this scenario? The replica was Online, healthy: true, with sufficient pool space. Something in resizeable_replicas() or the spec comparison logic excluded it.

Workaround used: Removed stale nexus child via REST API (DELETE /v0/nexuses/{nexus-id}/children/{url-encoded-child-uri}), then resize succeeded immediately.