snapshot restore functionality broken

[ngarafol]

Describe the bug
Every time I run "vault operator snapshot restore " command, cluster where I am restoring ends up with corrupted data and needs stopping each node, removing raft data, initializing of cluster, restoring snapshot again with cluster root token.

To Reproduce
Steps to reproduce the behavior:

1. On "origin" vault cluster run vault operator raft snapshot save snapshot-origin-cluster
2. On "destination" vault cluster run vault operator raft snapshot restore (-force) snapshot-origin-cluster
3. run vault status on destination vault cluster - no output
4. try to unseal nodes:

{"errors":["node was removed from a HA cluster"]}
{"errors":["node was removed from a HA cluster"]}
{"errors":["node was removed from a HA cluster"]}
{"errors":["node was removed from a HA cluster"]}
{"errors":["node was removed from a HA cluster"]}
{"errors":["node was removed from a HA cluster"]}

5. destination cluster is broken, fix it

Expected behaviorIf I am restoring snapshot from different cluster, I would expect destination cluster to seal since keys dont match.
If I am restoring snapshot from same cluster, I would expect destination cluster increasing raft index and staying unsealed

Environment:

• Vault Server Version (retrieve with vault status): 1.21.0
• Vault CLI Version (retrieve with vault version): v1.16.2
• Server Operating System/Architecture: Ubuntu 24.04, amd64

Vault server configuration file(s):

{
    "api_addr": "https://n02.example.org:443",
    "cluster_addr": "https://n02.example.org:444",
    "disable_mlock": false,
    "listener": [
        {
            "tcp": {
                "address": "0.0.0.0:443",
                "cluster_address": "0.0.0.0:444",
                "telemetry": [
                    {
                        "unauthenticated_metrics_access": true
                    }
                ],
                "tls_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                "tls_client_ca_file": "/etc/ssl/certs/ca-certificates.crt",
                "tls_disable": false,
                "tls_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
            }
        }
    ],
    "log_level": "Error",
    "storage": {
        "raft": {
            "path": "/etc/vault.d/raft",
            "performance_multiplier": "1",
            "retry_join": [
                {
                    "leader_api_addr": "https://n01.example.org",
                    "leader_ca_cert_file": "/etc/ssl/certs/ca-certificates.crt",
                    "leader_client_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                    "leader_client_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
                },
                {
                    "leader_api_addr": "https://n02.example.org",
                    "leader_ca_cert_file": "/etc/ssl/certs/ca-certificates.crt",
                    "leader_client_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                    "leader_client_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
                },
                {
                    "leader_api_addr": "https://n03.example.org",
                    "leader_ca_cert_file": "/etc/ssl/certs/ca-certificates.crt",
                    "leader_client_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                    "leader_client_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
                },
                {
                    "leader_api_addr": "https://n04.example.org",
                    "leader_ca_cert_file": "/etc/ssl/certs/ca-certificates.crt",
                    "leader_client_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                    "leader_client_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
                },
                {
                    "leader_api_addr": "https://n05.example.org",
                    "leader_ca_cert_file": "/etc/ssl/certs/ca-certificates.crt",
                    "leader_client_cert_file": "/etc/letsencrypt/live/n02.example.org/fullchain.pem",
                    "leader_client_key_file": "/etc/letsencrypt/live/n02.example.org/privkey.pem"
                }
            ]
        }
    },
    "telemetry": {
        "disable_hostname": true,
        "prometheus_retention_time": "30s"
    },
    "ui": false
}

Additional context
Before upgrading to 1.21.0 I used 1.16.0 and I had no issues with restoring with or without force option. It literally worked as expected behaviour. So, I assume somewhere between 1.16.0 and 1.21.0 snapshot restore got broken. Broken, because in my humble opinion it makes no sense if one restores SAME cluster data, that destination where snapshot is restored gets broken, needs raft data removed, cluster reinitialized and restored with force again...

[dosubot]

Thanks for the detailed report, @ngarafol. A few things to look into based on what you're describing:

CLI / Server version mismatch — You noted your server is v1.21.0 but your CLI is v1.16.2. Vault source and target versions should match for snapshot restore operations, and a significant version mismatch between CLI and server could contribute to unexpected behavior. I'd recommend upgrading your CLI to match the server version.

Cross-cluster communication during restore — The most common cause of the behavior you're describing (cluster "breaks" after restore, requires removing raft data and reinitializing) is that follower nodes attempt to rejoin the source cluster instead of the restored cluster when both clusters can communicate over the network. This prevents quorum from forming and effectively corrupts the destination cluster's state. The fix is to ensure full network isolation of the destination cluster before restoring — block all inbound/outbound traffic to other Vault clusters, remove it from load balancers, and ideally place it on a separate subnet (support article on this).

Same-cluster restore — Even when restoring to the same cluster, if you're using the -force flag, Vault replaces the Raft data entirely. After restore, the cluster will seal and you'll need to unseal it with the original unseal keys from the snapshot. Without -force, the restore should work if the seal keys match. A documentation update recently clarified that snapshot restore does not automatically unseal Vault — manual unsealing is required afterward.

Recommended restore procedure:

1. Isolate the destination cluster from all other Vault clusters (firewall rules, security groups, etc.)
2. Upgrade your CLI to match the server version (v1.21.0)
3. Run vault operator raft snapshot restore -force <snapshot_file> against the leader node
4. Unseal each node with vault operator unseal using the unseal keys from the cluster that originally created the snapshot

Could you clarify:

• Are you restoring between different clusters or the same cluster?
• Are the source and destination clusters able to communicate with each other over the network during the restore?
• What specific errors or symptoms do you see after the restore (e.g., error messages in vault status or the server logs)?

This would help narrow down whether this is the cross-cluster communication issue or something else.

Written by @dosu, approved by heatherezell

To reply, just mention @dosu.

---

Docs are dead. Just use Dosu.

[ngarafol]

Forgot to comment...

I am restoring between different clusters, yes. I upgraded CLI, so now it matches server version. But I failed to block connectivity between clusters.
So, destination and source clusters are currently able to communicate during restore.
And the log messages are the same as mentioned above, all destination cluster (where I am trying to restore) nodes are removed from HA cluster.

I need to block connectivity and try again.

[ngarafol]

Seems that blocking connectivity between clusters (just vault ports used) seems to be enough for successful restore. Thanks for suggestions, and sorry to keep you waiting.