Support snapshot auto-rebuilding

[everythings-gonna-be-alright]

Is your feature request related to a problem? Please describe.
When some of the pools suddenly change status to offline, Mayastor loses the ability to create new disks from existing snapshots if part of those snapshots are stored on an offline pool. Even if a snapshot has a replication factor of 2, and an online replica of that snapshot is available on another online pool.

Describe the solution you'd like

• Add logic to automatically rebuild a snapshot on another available pool if the original pool changes its status to offline, and if we have other available online pools with enough free disk space
• Allow to create disks from only one available snapshot replica

Additional context
The original disk was deleted. In this case, we have only a snapshot.

~ λ kubectl mayastor get volume-snapshots|grep -e 'ID' -e 'c14c8b3a-a966-4924-aa07-ed9835ff8307'
 ID                                    TIMESTAMP             SOURCE-SIZE  ALLOCATED-SIZE  TOTAL-ALLOCATED-SIZE  SOURCE-VOL                            RESTORES  SNAPSHOT_REPLICAS
 c14c8b3a-a966-4924-aa07-ed9835ff8307  2025-08-08T21:35:37Z  50 GiB       50 GiB          50 GiB                dea0d146-cddf-4569-a9fa-6adbe2f7e40b  5         2

~ λ kubectl mayastor get volumes|grep 'dea0d146-cddf-4569-a9fa-6adbe2f7e40b'
~ λ

This will be useful in cases where a user has a golden image (snapshot) and wants to create new disks from that existing golden image (snapshot).

[tiagolobocastro]

I had started on the snapshot rebuild, prior to the OEP process, it's quite a large list and here's the extended list of tasks:

✅ Done

• [data-plane/spdk] Add get_cluster_bitmap API to openebs/spdk
• [data-plane/api] Add ReplicaSnapshotRebuild job proto
• [data-plane] Add ReplicaSnapshotRebuild job dummy impls
• [data-plane] Add initial ReplicaSnapshotRebuild job
• [control-plane] Implement ReplicaSnapshotRebuild gRPC
• [data-plane/cli] Add snapshot rebuild CLI subcommand
• Update rebuild API to support push model rebuild

📝 To Do

• [control-plane] Implement Replica Snapshot Rebuild Reconciler (Paused)
• Rebuild snapshot chain index when loading pre-chain snapshot
• Add gRPC API to whitelist snapshot
• Implement gRPC snapshot whitelist
• Implement gRPC snapshot whitelist (control-plane)
• Implement reset replica snapshot
• Add gRPC API to fetch cluster bitmap
• Export cluster bitmap via gRPC
• Export/Import cluster bitmap via NVMe-admin command
• Retrieve cluster bitmap from CLI
• Call get_cluster_bitmap from SPDK
• Get cluster bitmap as diff from Snapshot to Replica
• Changes for NexusRebuild job (API)
• Changes for NexusRebuild job
• Add chain index to gRPC proto
• Impl dummy chain index on gRPC reply
• Parse snapshot chain index request and reply
• Impl dummy chain index for data-plane API
• Add snapshot chain index logic
• Determine replica is synced to the snapshot chain
• Determine if snapshot chain is fully synced across replicas
• Notify control-plane of Replica Snapshot Rebuild completion
• Handle async notification of Replica Snapshot Rebuild completion
• Implement Replica Snapshot Rebuild Progress Persistence
• Add replica snapshot to volume snapshot
• Implement SnapChain Rebuild Reconciler
• Add SnapRebuilding replica to Nexus as Inactive
• Don't allow snapshots if the snapshot tree is not in sync
• Garbage collect ReplicaSnapshot Rebuild
• Garbage collect SnapChain Rebuild
• Notify control-plane proto
• Add BDD feature and tests for the replica snapshot rebuild
• Add BDD feature and tests for snapshot rebuild
• Add get cluster bitmap diff API
• Use Seek API to fetch cluster bitmap (SPDK)
• Use Seek API to fetch cluster bitmap
• Flush IO as part of cluster bitmap IO freeze
• Copy only snapshot cluster diff on ReplicaSnapshotRebuild job
• Initial SnapChain Rebuild Reconciler

I'll try to raise an OEP with the design for this when I have some free time

Allow to create disks from only one available snapshot replica

This you should be able to already, if you specify a StorageClass of 1 replica? Then you just scale the volume up

[everythings-gonna-be-alright]

Oh, it looks like a really huge amount of work.
Thanks for the feedback.
Yep, it works if I reduce the number of snapshot replicas to 1.

Also, I found out one strange behaviour pattern.
As I can see, if the node with the snapshot root image goes down, Mayastor starts rebuilding each volume created from a snapshot without data deduplication. As a result, every volume created from a snapshot starts to consume space as a dedicated new volume, which leads to the consumption of all free disk space in my case.
It sounds difficult, but it would be great to implement logic that can recreate the root snapshot image first on a new node and then sync only the difference between the root and its child instead of creating a full image.

As I can see from the previous message, this logic has already been added to the development roadmap.

[everythings-gonna-be-alright]

Even if I create a snapshot from a volume with 4 replicas, I still get a full-cloned new replica on the node where I actually previously snapped a replica ( pool-on-node-42 ):

~ λ kubectl mayastor get volume-snapshots
 ID                                    TIMESTAMP             SOURCE-SIZE  ALLOCATED-SIZE  TOTAL-ALLOCATED-SIZE  SOURCE-VOL                            RESTORES  SNAPSHOT_REPLICAS
 9c567bd6-9a25-4047-bbcf-031d467edfa4  2025-11-10T22:59:03Z  4.7 GiB      4.7 GiB         4.7 GiB               0aeb071e-0e80-42bb-8c86-13bd216c9ba0  1         4

~ λ kubectl mayastor get volume fddfdda2-7df3-4b1f-8e41-0188bbfa5245
 ID                                    REPLICAS  TARGET-NODE             ACCESSIBILITY  STATUS  SIZE     THIN-PROVISIONED  ALLOCATED  SNAPSHOTS  SOURCE    CLEAN-SHUTDOWN  ENCRYPTED
 fddfdda2-7df3-4b1f-8e41-0188bbfa5245  2         worker-36               nvmf           Online  4.7 GiB  true              4.7 GiB    0          Snapshot  false           false

~ λ kubectl mayastor scale volume fddfdda2-7df3-4b1f-8e41-0188bbfa5245 3

~ λ kubectl mayastor get volume-replica-topology fddfdda2-7df3-4b1f-8e41-0188bbfa5245
 ID                                    NODE                    POOL             STATUS  ENCRYPTED  CAPACITY  ALLOCATED  SNAPSHOTS  CHILD-STATUS  REASON  REBUILD  HEALTHY
 d149f568-8d1e-4f67-89b6-c74ef2a7e4e5  worker-21               pool-on-node-21  Online  false      4.7 GiB   12 MiB     0 B        Online        <none>  <none>   true
 d8ad0b6d-a449-4709-98fb-1fe14e96ce54  worker-36               pool-on-node-36  Online  false      4.7 GiB   12 MiB     0 B        Online        <none>  <none>   true
 dcf3fe5e-31c4-466e-990b-c4131a51fd8e  worker-42               pool-on-node-42  Online  false      4.7 GiB   4.7 GiB    0 B        Online        <none>  <none>   true

[tiagolobocastro]

So you took a snapshot of 4-r volume 0aeb071e-0e80-42bb-8c86-13bd216c9ba0
Then you created a 2-r restore from snapshot
Then you scaled up the restored 2-r volume to a 3-r volume

hmm I think yes in this case we're not using the snapshot on the same node because we see it here as a new replica; and it may even come from a different pool
I don't think we had this use-case even for the snapshot rebuild tasks above, but this seems worth considering and doable once we have the bitmap rebuild since we have the common dependency tree