[AVM Module Issue]: Unable to delete VM associated with Immutable recovery services vault

[robsissons-contino]

Check for previous/existing GitHub issues

• I have checked for previous/existing GitHub issues

Issue Type?

I'm not sure

(Optional) Module Version

No response

(Optional) Correlation Id

No response

Description

When we have a recovery services vault with immutability configured (unlocked) and use the AVM to associate the VM with a policy on the vault we are unable to remove the VM from our Terraform configuration whilst there are recovery points available in the RSV.

Error:

deleting Resource: (ResourceId
"/subscriptions/***/resourceGroups/rg-xxx/providers/Microsoft.RecoveryServices/vaults/rsv-xxx/backupFabrics/Azure/protectionContainers/iaasvmcontainer;iaasvmcontainerv2;rg-xxxx;vm-xxxx/protectedItems/VM;iaasvmcontainerv2;rg-xxx;vm-xxxx"
/ Api Version "2024-10-01"): GET
https://management.azure.com/subscriptions/***/resourceGroups/rg-xxx/providers/Microsoft.RecoveryServices/vaults/RSV-xxx/backupOperations/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: UserErrorDeleteBackupItemOnImmutableItem
--------------------------------------------------------------------------------
{
  "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx",
  "name": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx",
  "status": "Failed",
  "startTime": "2025-09-18T14:22:18.3983532Z",
  "endTime": "2025-09-18T14:22:18.3983532Z",
  "error": {
    "code": "UserErrorDeleteBackupItemOnImmutableItem",
    "message": "The Backup Item(s) cannot be deleted since it has active recovery points and the selected Vault is an immutable vault."
  }
}
--------------------------------------------------------------------------------

Confirmed with Microsoft support that deletion of a virtual machine with recovery points in an RSV is possible. Also tried disabling the backup protection using AZ CLI and re-running - same error.

Finally tested by removing the VM through the portal to confirm that this should be possible and can confirm that the VM can be deleted and the recovery points remain available in the immutable vault.

Has anyone else faced this issue?

[robsissons-contino]

Anyone else seeing this issue? No response since logging in September

[lonegunmanb]

Hi @robsissons-contino, thank you for reporting this issue and for the detailed investigation — the error details and your testing (including portal deletion and AZ CLI attempts) were very helpful in pinpointing the root cause.

Root Cause Analysis

This issue stems from the module's migration of the backup protection resource from azurerm_backup_protected_vm to azapi_resource. The two behave very differently during terraform destroy:

azurerm_backup_protected_vm (previous implementation):
The AzureRM provider's Delete function has a feature flag — vm_backup_stop_protection_and_retain_data_on_destroy. When enabled, instead of sending a standard HTTP DELETE (which tries to delete the backup item and its recovery points), it sends a PUT request that sets ProtectionState to ProtectionStopped. This stops future backups but preserves all existing recovery points in the vault. Terraform then simply removes the resource from state.

azapi_resource (current implementation):
azapi_resource is a generic provider that maps Terraform operations directly to Azure REST API calls. On destroy, it always sends an HTTP DELETE request. There is no mechanism to customize the delete behavior — no delete_body, no delete_method override, and no skip_delete option (ref: azapi_resource docs). When the target vault has immutability enabled and active recovery points exist, Azure rejects the DELETE with UserErrorDeleteBackupItemOnImmutableItem — and this is a permanent rejection, not a transient error.

Why portal deletion works

When you delete a VM through the portal, Azure deletes the VM itself but does not delete the backup protection item in the vault. The recovery points remain available. This is essentially the same behavior as the AzureRM provider's "stop protection and retain data" approach.

Proposed Fix

We are evaluating reverting the backup resource from azapi_resource back to azurerm_backup_protected_vm. This would allow users with immutable vaults to configure the provider's feature flag:

provider "azurerm" {
  features {
    recovery_service {
      vm_backup_stop_protection_and_retain_data_on_destroy = true
    }
  }
}

This is the most reliable approach because it leverages the AzureRM provider's native, well-tested delete handling for this specific scenario. The commented-out azurerm_backup_protected_vm code is still present in the module's main.backup.tf, and a moved block already exists, so the migration path is straightforward.

In the meantime, as a workaround, you can manually stop protection before destroying:

# Stop protection but retain recovery points
az backup protection disable \
  --resource-group <vault-rg> \
  --vault-name <vault-name> \
  --container-name "iaasvmcontainer;iaasvmcontainerv2;<vm-rg>;<vm-name>" \
  --item-name "VM;iaasvmcontainerv2;<vm-rg>;<vm-name>" \
  --retain-data true

# Remove from Terraform state
terraform state rm 'module.<name>.azapi_resource.this_backup_intent["<key>"]'

# Then destroy normally
terraform destroy

We'll update this issue as the fix progresses.