Cannot deploy on Private Network: Failed to create Aml RP virtual workspace due to System.Exception: Failed async operation

[DarylGraves]

I am trying to follow the code here to deploy AI Foundry in a private network. It doesn't work.

I have tried this on two subscriptions:

• A work subscription where a Vnet has already been set up so I updated the parameters accordingly.
• My own personal subscription which was completely empty before this deployment - I didn't change nay of the parameters for this.

Both seem to fail on the same modules:

The error I receive is as follows:

{
  "code": "DeploymentFailed",
  "target": "/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.Resources/deployments/aiservicesznbv-znbv-deployment",
  "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
  "details": [
    {
      "code": "ResourceDeploymentFailure",
      "target": "/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.CognitiveServices/accounts/aiservicesznbv",
      "message": "The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'."
    }
  ]
}

If I go to the Activity Logs I get more info:

"properties": {
        "statusCode": "OK",
        "statusMessage": "\"{\\\"provisioningState\\\":\\\"Failed\\\",\\\"error\\\":\\\"{\\\\\\\"code\\\\\\\":\\\\\\\"DeploymentFailed\\\\\\\",\\\\\\\"target\\\\\\\":\\\\\\\"/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.Resources/deployments/aiFoundry-od4beeqvyz2yi\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\\\\\",\\\\\\\"details\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"ResourceDeploymentFailure\\\\\\\",\\\\\\\"target\\\\\\\":\\\\\\\"/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.Resources/deployments/aiservicesznbv-znbv-deployment\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\\\\\",\\\\\\\"details\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"DeploymentFailed\\\\\\\",\\\\\\\"target\\\\\\\":\\\\\\\"/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.Resources/deployments/aiservicesznbv-znbv-deployment\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\\\\\",\\\\\\\"details\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"ResourceDeploymentFailure\\\\\\\",\\\\\\\"target\\\\\\\":\\\\\\\"/subscriptions/redacted/resourceGroups/Test-AiFoundry/providers/Microsoft.CognitiveServices/accounts/aiservicesznbv\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\\\\\",\\\\\\\"details\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"OperationError\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"Failed to Create the resource. Provisioning state: Failed\\\\\\\",\\\\\\\"details\\\\\\\":[{\\\\\\\"code\\\\\\\":\\\\\\\"ResourceProviderError\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"Failed to create Aml RP virtual workspace due to System.Exception: Failed async operation {\\\\\\\\n  \\\\\\\\\\\\\\\"status\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"Failed\\\\\\\\\\\\\\\",\\\\\\\\n  \\\\\\\\\\\\\\\"error\\\\\\\\\\\\\\\": {\\\\\\\\n    \\\\\\\\\\\\\\\"code\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"InternalServerError\\\\\\\\\\\\\\\",\\\\\\\\n    \\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\": \\\\\\\\\\\\\\\"Object reference not set to an instance of an object.\\\\\\\\\\\\\\\"\\\\\\\\n  }\\\\\\\\n}\\\\\\\\n   at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AsyncOperationHelper.PollUntilCompleteAsync[T](TraceContext traceContext, HttpResponseMessage response, HttpClient httpClient, Int32 timeoutInMin, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AsyncOperationHelper.cs:line 117\\\\\\\\n   at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AmlRpClient.PutAmlRpVirtualHubAsync(TraceContext traceContext, Dictionary`2 headers, ResourceIdentityInArm resourceIdentity, String primaryUserAssignedIdentity, String internalId, KeyVaultProperties keyVaultProperties, String subscriptionId, String resourceGroupName, String workspaceName, List`1 networkInjections, PublicNetworkAccessType publicNetworkAccessType, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AmlRpClient.cs:line 119\\\\\\\\n   at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AmlRpService.PutAmlRpWorkspaceAsync(TraceContext traceContext, ResourceRequestContext requestContext, WorkerMessageRequestContext workerMessageRequestContext, IResourceEntity insertedResourceEntity, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AmlRpService.cs:line 124\\\\\\\\n   at Microsoft.CognitiveServices.ResourceProvider.Worker.MessageProcessor.ProcessInternalAsync(TraceContext traceContext, RpWorkerQueueMessage message, CancellationToken cancellationToken) in /__w/1/s/src/ResourceProvider/Rp.WorkerRole/AccountProvisioning/MessageProcessor.cs:line 223\\\\\\\"}]}]}]}]}]}\\\"}\"",

Any help would be most appreciated, thank you!

[leestott]

@fosteramanda please see this issue

Here is a similar issue: #280

And this one, same error message, #308

[leestott]

Hi @DarylGraves

From the error trace:
"Failed to create Aml RP virtual workspace due to System.Exception: Object reference not set to an instance of an object"

This typically means:

• The AML workspace creation failed due to a null reference in the RP.
• Most likely triggered by missing or misconfigured network injection parameters, user-assigned identity, or Key Vault linkage.

Here’s a checklist to validate against your deployment:

User Assigned Identity:

• Ensure it's created and assigned correctly.
• Must have Contributor or Owner role on the AML workspace and Key Vault.

Key Vault Properties:

• Ensure keyVaultProperties are passed correctly to AML RP.
• Key Vault must allow access from the AML workspace identity.

Network Injection:

• Validate that networkInjections include correct subnet IDs.
• Subnets must have Microsoft.MachineLearningServices/workspaces delegation.

Workspace Name & Location:

• Must match supported regions for AML + Cognitive Services.
• Use az ml workspace list-keys to validate access post-deployment.

Public Network Access:

• If set to Disabled, ensure all dependencies (Key Vault, DNS, etc.) are reachable via private endpoints.

Validate using the following

1. User Assigned Identity

Check the userAssignedIdentities block in the template:

"identity": {
  "type": "UserAssigned",
  "userAssignedIdentities": {
    "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('identityName'))]": {}
  }
}

Ensure:

• The identity exists before AML workspace creation.
• It has Contributor or Owner role on:
  • AML workspace
  • Key Vault
  • Subnet (if using private endpoints)

2. Key Vault Properties

This is passed to AML RP via keyVaultProperties:

"keyVaultProperties": {
  "keyVaultArmId": "[resourceId('Microsoft.KeyVault/vaults', parameters('keyVaultName'))]",
  "keyIdentifier": "[parameters('keyIdentifier')]"
}

Ensure:

• keyVaultArmId points to a valid Key Vault.
• keyIdentifier is a valid key URI (e.g., https://yourvault.vault.azure.net/keys/yourkey/123456)
• The Key Vault allows access from the AML workspace identity:
  • Use accessPolicies or RBAC depending on your setup.

3. Network Injection

This is where things often break. Check the networkInjections array:

"networkInjections": [
  {
    "subnetId": "[parameters('subnetId')]",
    "roleAssignments": [
      {
        "principalId": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('identityName')), '2021-09-30').principalId]",
        "roleDefinitionId": "[variables('contributorRoleId')]"
      }
    ]
  }
]

Ensure:

• subnetId points to a subnet with delegation for Microsoft.MachineLearningServices/workspaces.
• The subnet is in the same region as the AML workspace.
• NSGs and route tables allow outbound traffic to required endpoints (Key Vault, Storage, etc.).

---

4. Public Network Access

If you're disabling public access:

"publicNetworkAccess": "Disabled"

Ensure:

• All dependencies (Key Vault, Storage, DNS) are reachable via private endpoints.
• DNS zones are correctly linked and resolvable inside the VNet.

Debug Tip

Add outputs to your template to trace values:

"outputs": {
  "workspaceName": {
    "type": "string",
    "value": "[parameters('workspaceName')]"
  },
  "identityPrincipalId": {
    "type": "string",
    "value": "[reference(resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', parameters('identityName')), '2021-09-30').principalId]"
  }
}

This helps confirm identity bindings and workspace parameters post-deployment.

Hope this helps unblock you.

[DarylGraves]

Hi,

This isn't a custom deployment - I was just using the template provided in the git repository so there is no user assigned identity or keyvault and public network access should be disabled as per the template.

You mentioned above that the subnet should have a delegation to Microsoft.MachineLearningServices/workspaces but in the README.md file it states Microsoft.App/environments

• On my personal subscription I do not change any parameters so the deployment should create all the Vnets, delegations and resources for me
• On my work subscription I tried to use it against an existing Vnet and Subnets which have the Microsoft.App/environments

Actually, it seems that this isn't actually related to the template. I am getting the same issue if I just run this:

// AI Foundry
param accountName string = 'aifoundry-uat
@description('The name of the model you want to deploy')
param modelName string = 'gpt-4o'
@description('The provider of your model')
param modelFormat string = 'OpenAI'
@description('The version of your model')
param modelVersion string = '2024-11-20'
@description('The sku of your model deployment')
param modelSkuName string = 'GlobalStandard'
@description('The tokens per minute (TPM) of your model deployment')
param modelCapacity int = 30

resource aiAccount 'Microsoft.CognitiveServices/accounts@2025-04-01-preview' = {
  name: toLower(accountName)
  location: location
  sku: {
    name: 'S0'
  }
  kind: 'AIServices'
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    allowProjectManagement: true
    customSubDomainName: toLower(accountName)
    networkAcls: {
      defaultAction: 'Allow'
      virtualNetworkRules: []
      ipRules: []
    }
    publicNetworkAccess: 'Disabled'
    networkInjections: [
      {
        scenario: 'agent'
        // subnetArmId: AgentSubnet.id
        subnetArmId: '/subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/virtualNetworks/redacted/subnets/redacted'
        useMicrosoftManagedNetwork: false
      }
    ]
    disableLocalAuth: false
  }
}

And the same even if I remove networkInjections altogether I get "Failed to create Aml RP virtual workspace due to System.Exception"

On a tangent, I also want to highlight your pipeline which pushes these Git Issues into Azure DevOps is not working:

[ukhan262]

#314 please check the comment on that one. It seems to be the same issue.

[DarylGraves]

#314 please check the comment on that one. It seems to be the same issue.

Thanks but I got the same error when I deployed a completely new instance in a fresh subscription so I don't believe this is the case. I've tried purging the ai foundries and many different names.

[ukhan262]

In my case, it worked this morning. I know that MSFT is actively making changes in their AI stream regardless of making it GA so its possible that it could be something else.

[Thialala]

Hi,
I have the same issue. If I deploy the template as is, I have the error.
For a workaround, I comment the networkInjections part. And the first deployment works. Then I uncomment, I redeploy again and it works.

[DarylGraves]

Because I requested early-access to create this on a Class A IP Range I had a contact at Microsoft I was able to turn to for help:

It turns out my error was because Microsoft.App was not registered against the subscription. I mistakenly assumed it was because we have deployed a number of App Service Plans in the same subscription but I guess this is for something else.

In my situation, I was able to delete the Resource Group, register the providers listed in the Prerequisites here and re-run the deployment. This now works.