Build fails with "HashConflictOnDifferentRoleAssignmentIds"

[tanya-borisova]

Main build sometimes fails with an intermittent issue:

ERROR    resources.resource:resource.py:78 Failed to deploy status message: 62ce9147-d327-44ed-b9ee-6b0deb79d135: Error context message = ╷ │ Error: authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="HashConflictOnDifferentRoleAssignmentIds" Message="A hash conflict was encountered for the role Assignment ID. Please use a new Guid." │  │   with module.airlock[0].azurerm_role_assignment.sa_import_approved_reader, │   on airlock/storage_accounts.tf line 294, in resource "azurerm_role_assignment" "sa_import_approved_reader": │  294: resource "azurerm_role_assignment" "sa_import_approved_reader" { │  ╵ Error: error running command /cnab/app/terraform terraform apply -auto-approve -input=false -var client_secret= -var address_space=10.2.139.0/24 -var app_role_id_workspace_owner= -var app_role_id_workspace_researcher= -var app_service_plan_sku=P1v3 -var auth_client_id=******* -var auth_client_secret=******* -var auth_tenant_id=******* -var client_id=auto_create -var aad_redirect_uris_b64=W10= -var enable_airlock=true -var enable_local_debugging=false -var location=*** -var register_aad_application=true -var scope_id= -var shared_storage_quota=50 -var sp_id= -var tre_id=*** -var tre_resource_id=62ce9147-d327-44ed-b9ee-6b0deb79d135 -var workspace_owner_object_id=ade81c51-edee-4f17-ac8d-1becb3dd7352: exit status 1 Error: mixin execution failed: exit status 1 Error: 2 errors occurred: 	* container exit code: 1, message: <nil>. fetching outputs failed: error copying outputs from container: Error: No such container:path: a827214dce7ccb5302fcc3e9599315ad99d36c07368ab8b6776eb1caf87500a2:/cnab/app/outputs 	* required output app_role_id_workspace_owner is missing and has no default   az login --identity -u 261b930f-39a8-4266-bf69-abdb98fddebe && az acr login --name ***mgmt && porter install "62ce9147-d327-44ed-b9ee-6b0deb79d135"  --reference ***mgmt.azurecr.io/tre-workspace-base:v0.3.19  --param address_space="10.2.139.0/24" --param arm_use_msi="true" --param azure_location="***" --param client_id="auto_create" --param id="62ce9147-d327-44ed-b9ee-6b0deb79d135" --param register_aad_application="True" --param tfstate_container_name="tfstate" --param tfstate_resource_group_name="rg-***-mgmt" --param tfstate_storage_account_name="***mgmt" --param tre_id="***" --param workspace_owner_object_id="ade81c51-edee-4f17-ac8d-1becb3dd7352" --allow-docker-host-access --force --cred ./vmss_porter/arm_auth_local_debugging.json --cred ./vmss_porter/aad_auth.json && porter show 62ce9147-d327-44ed-b9ee-6b0deb79d135

Example runs:

https://github.com/microsoft/AzureTRE/runs/7457454389?check_suite_focus=true
https://github.com/microsoft/AzureTRE/runs/7505818171?check_suite_focus=true

[tanya-borisova]

Known mitigation is to retry

[tanya-borisova]

Also see the corresponding issue in azurerm provider: hashicorp/terraform-provider-azurerm#17155

[stuartleeks]

I've attempted to get a reliable way to reproduce this error and created the terraform below (note, when running, add -parallelism=50 😄).

This creates container_count containers in a storage account and assigns the running user 4 different roles against each container.

I've run this multiple times with 150 containers and haven't seen the issue 😞

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.16.0"
    }
  }
}

provider "azurerm" {
  features {}
}

variable "rg_name" {}
variable "storage_name" {}

variable "container_count" {
	default = 150
	type = number
}

data "azurerm_client_config" "current" {
}

resource "azurerm_resource_group" "test" {
  location = "northeurope"
  name     = var.rg_name
}

resource "azurerm_role_assignment" "rg_contributor" {
  count                = 1
  scope                = azurerm_resource_group.test.id
  role_definition_name = "Contributor"
  principal_id         = data.azurerm_client_config.current.object_id

  depends_on = [
    azurerm_resource_group.test
  ]
}

resource "azurerm_storage_account" "test" {
  resource_group_name      = azurerm_resource_group.test.name
  location                 = azurerm_resource_group.test.location
  name                     = var.storage_name
  account_kind             = "StorageV2"
  account_tier             = "Standard"
  account_replication_type = "LRS"
}
resource "azurerm_storage_container" "example" {
  count                 = var.container_count
  name                  = "container-${format("%04d", count.index + 1)}"
  storage_account_name  = azurerm_storage_account.test.name
  container_access_type = "private"
}


resource "azurerm_role_assignment" "container_data_contributor" {
  count                = var.container_count
  scope                = "${azurerm_storage_account.test.id}/blobServices/default/containers/${azurerm_storage_container.example[count.index].name}"
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}
resource "azurerm_role_assignment" "container_data_reader" {
  count                = var.container_count
  scope                = "${azurerm_storage_account.test.id}/blobServices/default/containers/${azurerm_storage_container.example[count.index].name}"
  role_definition_name = "Storage Blob Data Reader"
  principal_id         = data.azurerm_client_config.current.object_id
}

resource "azurerm_role_assignment" "container_contributor" {
  count                = var.container_count
  scope                = "${azurerm_storage_account.test.id}/blobServices/default/containers/${azurerm_storage_container.example[count.index].name}"
  role_definition_name = "Contributor"
  principal_id         = data.azurerm_client_config.current.object_id
}
resource "azurerm_role_assignment" "container_reader" {
  count                = var.container_count
  scope                = "${azurerm_storage_account.test.id}/blobServices/default/containers/${azurerm_storage_container.example[count.index].name}"
  role_definition_name = "Reader"
  principal_id         = data.azurerm_client_config.current.object_id
}

[stuartleeks]

I've run this multiple times with 150 containers and haven't seen the issue 😞

... and immediately after this, I attempt to deploy a workspace and hit the issue!

Workspace creation is performed via porter (in a container) - could there be something about that environment that changes how GUIDs are generated??