Bug: [AKS] Cant update the cluster using 'createorupdate', the 'UserAssignedIdentities' property cant be parsed

[martijn1279]

Hello,

We have a AKS cluster with a user assigned managed identity but we cant update the cluster using the REST API or the Portal

Reproduction steps

1. create a aks cluster using Terraform:

1. make main.tf file with following config:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.46.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_user_assigned_identity" "example" {
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  name                = "example_user_assigned_identity"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "exampleaks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = azurerm_user_assigned_identity.example.id
  }
}

2. execute terraform init
3. execute terraform apply -auto-approve

2.1 updating using the Terraform

1. added following to the AKS resource in terraform:

api_server_authorized_ip_ranges = ["168.63.129.16"]

2. execute terraform apply -auto-approve -> error:

Error: updating Managed Kubernetes Cluster "example-aks1" (Resource Group "example-resources"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidIdentityValues" Message="Invalid value for the identities '/subscriptions/***/resourceGroups/example-resources/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example_user_assigned_identity'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."

  on main.tf line 25, in resource "azurerm_kubernetes_cluster" "example":
  25: resource "azurerm_kubernetes_cluster" "example" {

2.2 updating using the Rest API

If we want to update the aks cluster with 'createorupdate'(API version: 2021-02-01) rest call with the json obtained from the 'get'(API version: 2021-02-01) rest call we get the following error:

{
  "error": {
    "code": "InvalidIdentityValues",
    "message": "Invalid value for the identities '/subscriptions/***/resourceGroups/***/providers/Microsoft.ManagedIdentity/userAssignedIdentities/***'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."
  }
}

2.3 updating using the Azure Portal

if we want to set authorized IP ranges the aks cluster in portal we get the same error:

Thanks!

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm.

Issue Details

---

Hello,

We have a AKS cluster with a user assigned managed identity but we cant update the cluster using the REST API or the Portal

Reproduction steps

1. create a aks cluster using Terraform:

1. make main.tf file with following config:

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.46.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "example" {
  name     = "example-resources"
  location = "West Europe"
}

resource "azurerm_user_assigned_identity" "example" {
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  name                = "example_user_assigned_identity"
}

resource "azurerm_kubernetes_cluster" "example" {
  name                = "example-aks1"
  location            = azurerm_resource_group.example.location
  resource_group_name = azurerm_resource_group.example.name
  dns_prefix          = "exampleaks1"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  identity {
    type                      = "UserAssigned"
    user_assigned_identity_id = azurerm_user_assigned_identity.example.id
  }
}

2. execute terraform init
3. execute terraform apply -auto-approve

2.1 updating using the Terraform

1. added following to the AKS resource in terraform:

api_server_authorized_ip_ranges = "168.63.129.16"

2. execute terraform apply -auto-approve

2.2 updating using the Rest API

If we want to update the aks cluster with 'createorupdate'(API version: 2020-11-01) rest call with the json obtained from the 'get'(API version: 2020-11-01) rest call we get the following error:

{
  "error": {
    "code": "InvalidIdentityValues",
    "message": "Invalid value for the identities '/subscriptions/***/resourceGroups/***/providers/Microsoft.ManagedIdentity/userAssignedIdentities/***'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."
  }
}

2.3 updating using the Azure Portal

if we want to set authorized IP ranges the aks cluster in portal we get the same error:

Thanks!

Author:	martijn1279
Assignees:	ruowan
Labels:	AKS, Service Attention, question
Milestone:	-

[martijn1279]

I also can reproduce it if i use the Azure CLI:

# Create resource group
az group create `
   -l WestEurope `
   -n example-resources-cli

# create managed identity and get resource id
$id = az identity create --name example_user_assigned_identity-cli `
   --resource-group example-resources-cli `
   --query id

# create aks cluster with given managed identity
az aks create `
   -g example-resources-cli `
   -n example-aks-cli `
   --enable-managed-identity `
   --assign-identity $id `
   --no-ssh-key

[miwithro]

@martijn1279 I was not able to repro this scenario. I created a cluster with assigned MI, then upgraded the cluster to enable api server authorized IP ranges and it worked. I tested this from the portal, and from the Azure CLI. From the error messages it appears to be an issue with the MI. Can you verify the MI is created properly?

[martijn1279]

@miwithro if i use the CLI it works fine. But if i create a MI and a AKS cluster with the the CLI code below and then use the portal to enable api server authorized IP ranges i get the error.

az feature register --namespace Microsoft.ContainerService -n MigrateToMSIClusterPreview
az feature register --namespace Microsoft.ContainerService -n UserAssignedIdentityPreview

# Create resource group
az group create `
   -l WestEurope `
   -n example-resources-cli

# create managed identity and get resource id
$id = az identity create --name example_user_assigned_identity-cli `
   --resource-group example-resources-cli `
   --query id

# create aks cluster with given managed identity
az aks create `
   -g example-resources-cli `
   -n example-aks-cli `
   --enable-managed-identity `
   --assign-identity $id `
   --no-ssh-key

[miwithro]

@norshtein can you investigate?

[martijn1279]

any update?

[omichels]

can reproduce that exact problem