"Failed to create Aml RP virtual workspace due to System.Exception: Failed async operation", "Received 404 from a service request". - 15b-private-network-standard-agent-setup-byovnet EXAMPLE ERROR

[pitsnake69]

I am following the AI Foundry with VNET Injection directions and have successfully deployed my Terraform configuration into EastUS region. When I attempt to deploy into WestUS region I am receiving "Failed to create Aml RP virtual workspace due to System.Exception: Failed async operation", "Received 404 from a service request".

My working VNET in EASTUS is 172.18.0.0/22 with AgentHost subnet of 172.18.1.0/24 with Microsoft.Apps/environments service delegation;
Not working VNET in WESTUS is 172.18.24.0/22 with AgentHost subnet of 172.18.25.0/24 with Microsoft.Apps/environments service delegation;

Both VNET's use the same private DNS listener target for DNS resolution and have the same private DNS zone's linked, so DNS resolution is identical for both. I have triple checked everything matches up... resource providers, DNS, code, etc.. the only difference is the subscription and region. I have been spinning wheels here for several weeks and Microsoft Support is still working with me, but they appear stumped themselves currently. Has anyone that experienced this issue ever actually found a reason for it? Please do not link me to #309 as that is not a solution to this issue.

I am following private VNET injection example 15b and this is the resource creation that is failing; Full error is at the bottom.
(https://github.com/azure-ai-foundry/foundry-samples/blob/main/samples/microsoft/infrastructure-setup-terraform/15b-private-network-standard-agent-setup-byovnet/code/main.tf)

Create the AI Foundry resource

resource "azapi_resource" "ai_foundry" {
provider = azapi.workload_subscription

type = "Microsoft.CognitiveServices/accounts@2025-06-01"
name = "aifoundry${random_string.unique.result}"
parent_id = "/subscriptions/${var.subscription_id_resources}/resourceGroups/${var.resource_group_name_resources}"
location = var.location
schema_validation_enabled = false

body = {
kind = "AIServices",
sku = {
name = "S0"
}
identity = {
type = "SystemAssigned"
}

properties = {
  # Support both Entra ID and API Key authentication for underlining Cognitive Services account
  disableLocalAuth = false

  # Specifies that this is an AI Foundry resource
  allowProjectManagement = true

  # Set custom subdomain name for DNS names created for this Foundry resource
  customSubDomainName = "aifoundry${random_string.unique.result}"

  # Network-related controls
  # Disable public access but allow Trusted Azure Services exception
  publicNetworkAccess = "Disabled"
  networkAcls = {
    defaultAction = "Allow"
  }

  # Enable VNet injection for Standard Agents
  networkInjections = [
    {
      scenario                   = "agent"
      subnetArmId                = var.subnet_id_agent
      useMicrosoftManagedNetwork = false
    }
  ]
}

}
}

Error: Failed to create/update resource
with azapi_resource.exampleaifoundryname
on westus-azureaifoundry.tf line 128, in resource "azapi_resource" "exampleaifoundryname-001":
resource "azapi_resource" "exampleaifoundryname-001" {
creating/updating Resource: (ResourceId "/subscriptions//resourceGroups/rg-exampleaifoundryname-westus/providers/Microsoft.CognitiveServices/accounts/exampleaifoundryname-westus" / Api Version "2025-06-01"): GET https://management.azure.com/subscriptions//providers/Microsoft.CognitiveServices/locations/West US/operationResults/

RESPONSE 200: 200 OK
ERROR CODE: OperationError

{
"id": "/subscriptions//providers/Microsoft.CognitiveServices/locations/West US/operationResults/",
"name": "",
"status": "Failed",
"startTime": "2025-09-29",
"endTime": "2025-09-29",
"error": {
"code": "OperationError",
"message": "Failed to Create the resource. Provisioning state: Failed",
"details": [
{
"code": "ResourceProviderError",
"message": "Failed to create Aml RP virtual workspace due to System.Exception: Failed async operation {\n "status": "Failed",\n "error": {\n "code": "InternalServerError",\n "message": "Received 404 from a service request"\n }\n}\n at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AsyncOperationHelper.PollUntilCompleteAsync[T](TraceContext traceContext, HttpResponseMessage response, HttpClient httpClient, Int32 timeoutInMin, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AsyncOperationHelper.cs:line 117\n at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AmlRpClient.PutAmlRpVirtualHubAsync(TraceContext traceContext, Dictionary2 headers, ResourceIdentityInArm resourceIdentity, String primaryUserAssignedIdentity, String internalId, KeyVaultProperties keyVaultProperties, String subscriptionId, String resourceGroupName, String workspaceName, List1 networkInjections, PublicNetworkAccessType publicNetworkAccessType, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AmlRpClient.cs:line 119\n at Microsoft.CognitiveServices.ResourceProvider.AmlRp.AmlRpService.PutAmlRpWorkspaceAsync(TraceContext traceContext, ResourceRequestContext requestContext, WorkerMessageRequestContext workerMessageRequestContext, IResourceEntity insertedResourceEntity, CancellationToken cancellationToken) in /__w/1/s/src/Common/ResourceProvider/AmlRp/AmlRpService.cs:line 124\n at Microsoft.CognitiveServices.ResourceProvider.Worker.MessageProcessor.ProcessInternalAsync(TraceContext traceContext, RpWorkerQueueMessage message, CancellationToken cancellationToken) in /__w/1/s/src/ResourceProvider/Rp.WorkerRole/AccountProvisioning/MessageProcessor.cs:line 223"
}
]
}
}

[ukhan262]

Apparently, the issue is that you must purge all the AI foundry instances whether they were successful or in failed state regardless of the name of the AI foundry. The AI agent subnet holds on to it in the background that we do not see and after the purge we must wait for at least an hour or so regardless of the name of the AI foundry whether random or repeated. It is good to use a new name for the AI foundry.

[DarylGraves]

For my error (#309) it turned out it was because Microsoft.App was not registered against the subscription. I mistakenly assumed it was because we have deployed a number of App Service Plans in the same subscription but I guess this is for something else.

In my situation, I was able to delete the Resource Group, register the providers listed in the Prerequisites here and re-run the deployment with a different name. This now works for me. Hope this maybe helps.

[pitsnake69]

Thank you for the feedback. I have ruled out resource providers missing in my case, so still hunting for a solution.

…

On Wed, Oct 1, 2025 at 8:23 AM Daryl Graves ***@***.***> wrote: *DarylGraves* left a comment (azure-ai-foundry/foundry-samples#314) <#314 (comment)> For my error (#309 <#309>) it turned out it was because Microsoft.App was not registered against the subscription. I mistakenly assumed it was because we have deployed a number of App Service Plans in the same subscription but I guess this is for something else. In my situation, I was able to delete the Resource Group, register the providers listed in the Prerequisites here <https://github.com/azure-ai-foundry/foundry-samples/blob/main/samples/microsoft/infrastructure-setup/15-private-network-standard-agent-setup/README.md#prerequisites> and re-run the deployment with a different name. This now works for me. Hope this maybe helps. — Reply to this email directly, view it on GitHub <#314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXUIPCBVSJKGQTSUBLZEYL3VPBUXAVCNFSM6AAAAACH2ALOF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNJWGA3DINBRGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[pitsnake69]

MS Support had me try to purge the failed deployment and then completely delete the subnet and recreate subnet and then retry deployment. I ended up with the same error. I will try again today and wait 2 hours after purge and deletion of subnet before I attempt to recreate.

…

On Tue, Sep 30, 2025 at 12:52 PM Umar Khan ***@***.***> wrote: *ukhan262* left a comment (azure-ai-foundry/foundry-samples#314) <#314 (comment)> Apparently, the issue is that you must purge all the AI foundry instances whether they were successful or in failed state regardless of the name of the AI foundry. The AI agent subnet holds on to it in the background that we do not see and after the purge we must wait for at least an hour or so regardless of the name of the AI foundry whether random or repeated. It is good to use a new name for the AI foundry. — Reply to this email directly, view it on GitHub <#314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHXUIPA34QEFOXKNQIDT5XT3VKYLBAVCNFSM6AAAAACH2ALOF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNJTGAZTKMBZGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[DarylGraves]

Something else that was mentioned to me be Microsoft is the "Capability Host" apparently saw a previous deployment had failed and failed a new attempt. They recommended I tried to delete this too.

In my case, I just deleted the resource group, purged the AI Foundry from the AI Foundry section in Azure and that seemed to be sufficient, but they do provide a shell script to delete the Capability Host if you need to, so you could try running that too to be extra safe: https://github.com/azure-ai-foundry/foundry-samples/blob/main/samples/microsoft/infrastructure-setup/15-private-network-standard-agent-setup/deleteCapHost.sh

[ukhan262]

When you purge an AI foundry instance, please also delete the subnet and get that recreated and ensure you are not getting this error.

Failed to delete subnet. Error: The subnet is in use by a service association link (legionservicelink) and cannot be deleted. In order to delete the subnet, delete or detach all the resources associated with it. See aka.ms/deletesubnet.

If the above error comes then that means that the subnet is still bound to AI foundry and the best thing to do is retry the deletion after half an hour and then recreate the subnet and the AI foundry with private networking etc.