[BUG] Restore operation for rolling back Rancher from v2.13.0 to v2.12.3 never completes

[prachidamle]

Rancher Server Setup

• Rancher version: 2.12.3

Describe the bug

When performing a rollback from Rancher v2.13.0 to Rancher v2.12.3 using the Backup&Restore Operator, the restore process does not complete successfully.

BRO logs seen:

ERRO[2025/11/17 16:26:58] Error restoring cluster-scoped resources [error restoring cattle-globalrole-user-base of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-users-manage of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-user of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner error restoring cattle-globalrole-clusters-create of type rbac.authorization.k8s.io/v1, Resource=clusterroles: restoreResource: err updating resource admission webhook "rancher.cattle.io.clusterroles.rbac.authorization.k8s.io" denied the request: cannot modify or remove label authz.management.cattle.io/gr-owner]
ERRO[2025/11/17 16:26:58] error syncing 'restore-migration': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing

Expected behavior
Restore should finish within reasonable time without errors.

[prachidamle]

This issue needs to be solved by orchestrating webhook deployment during the restore runs. Details here: #833

For v2.13.0, there is a workaround found by @jbiers to let the restore complete.

[jbiers]

Resolution

The bug will be officially fixed in the v2.14.0 version of Rancher as part of improvements tracked by this Github issue. There is, however, a workaround solution to prevent this bug from happening.

Disabling the Rancher Webhook during a Restore

The official documentation on how to perform a rollback can be found here. Essentially, the usual steps will look like this (summarized from the documentation):

1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3.
2. Create a BRO Restore CR referencing the desired Backup, wait for it to be completed.
3. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system

To prevent the bug from happening, two extra steps are needed (points 2 and 3):

1. In a cluster running Rancher v2.13.0, make sure you have access to a Backup taken for Rancher v2.12.3.
2. Scale down Rancher to 0 replicas kubectl scale deploy/rancher -n cattle-system --replicas=0
3. Uninstall the rancher webhook with helm uninstall helm uninstall rancher-webhook -n cattle-system. Make sure it was uninstalled correctly with helm list -n cattle-system.
4. Create a BRO Restore CR (via kubectl, as Rancher is down) referencing the desired Backup, wait for it to be completed.
5. Perform the Helm Rollback as referenced in the docs, usually helm rollback rancher -n cattle-system

[LucasSaintarbor]

@prachidamle @jbiers Added a RN for this issue via 47fa469. Thanks!