404 Resource Not Found when creating express route cross connection private peering.

[wizzfizz94]

Describe the bug
Receive a 404 resource not found error when trying to create a cross connection peering using beginCreateOrUpdateAsync and also createOrUpdate can confirm same details work via cURL.

Exception or Stack Trace

{
    "error": {
        "code": "NotFound",
        "message": "Resource /subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering not found.",
        "details": []
    }
}

To Reproduce

1. Created expressRouteCrossConnectionPeeringInner with peerAsn, primaryPeerAddressPrefix, secondaryPeerAddressPrefix, vlanId configured.
2. make the sdk call
3. 404 result above

Code Snippet

 azure.networks().manager().serviceClient().getExpressRouteCrossConnectionPeerings()
        .createOrUpdate(
          resourceGroupName,
          serviceKey,
          peeringType,
          expressRouteCrossConnectionPeeringInner,
          com.azure.core.util.Context.NONE
        )

Expected behavior
2xx response

Setup (please complete the following information):

• OS: MAC OSX
• IDE: vscode
• Library/Libraries:

+- com.azure.resourcemanager:azure-resourcemanager:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-resources:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-storage:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-compute:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-network:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-keyvault:jar:2.48.0:compile
[INFO] |  |  +- com.azure:azure-security-keyvault-keys:jar:4.9.2:compile
[INFO] |  |  \- com.azure:azure-security-keyvault-secrets:jar:4.9.2:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-msi:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-sql:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-authorization:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-appservice:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-cosmos:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-containerservice:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-monitor:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-containerregistry:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-dns:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-appplatform:jar:2.48.0:compile
[INFO] |  |  \- com.azure:azure-storage-file-share:jar:12.25.0:compile
[INFO] |  |     \- com.azure:azure-storage-common:jar:12.28.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-containerinstance:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-privatedns:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-redis:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-eventhubs:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-trafficmanager:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-servicebus:jar:2.48.0:compile
[INFO] |  +- com.azure.resourcemanager:azure-resourcemanager-cdn:jar:2.48.0:compile
[INFO] |  \- com.azure.resourcemanager:azure-resourcemanager-search:jar:2.48.0:compile
[INFO] +- com.azure:azure-identity:jar:1.15.0:compile
[INFO] |  +- com.azure:azure-json:jar:1.3.0:compile
[INFO] |  +- com.microsoft.azure:msal4j:jar:1.17.2:compile
[INFO] |  |  +- com.nimbusds:oauth2-oidc-sdk:jar:7.1.1:compile
[INFO] |  |  |  +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] |  |  |  +- com.nimbusds:content-type:jar:2.0:compile
[INFO] |  |  |  +- com.nimbusds:lang-tag:jar:1.4.4:compile
[INFO] |  |  |  +- com.nimbusds:nimbus-jose-jwt:jar:8.19:compile
[INFO] |  |  |  \- com.sun.mail:javax.mail:jar:1.6.1:compile
[INFO] |  |  |     \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  \- net.minidev:json-smart:jar:2.3:compile
[INFO] |  |     \- net.minidev:accessors-smart:jar:1.2:compile
[INFO] |  |        \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] |  +- com.microsoft.azure:msal4j-persistence-extension:jar:1.3.0:compile
[INFO] |  |  \- net.java.dev.jna:jna:jar:5.13.0:compile
[INFO] |  \- net.java.dev.jna:jna-platform:jar:5.13.0:compile
[INFO] +- com.azure:azure-core-http-netty:jar:1.9.0:compile
[INFO] |  +- io.netty:netty-handler:jar:4.1.58.Final:compile
[INFO] |  |  +- io.netty:netty-common:jar:4.1.58.Final:compile
[INFO] |  |  +- io.netty:netty-resolver:jar:4.1.58.Final:compile
[INFO] |  |  +- io.netty:netty-transport:jar:4.1.58.Final:compile
[INFO] |  |  \- io.netty:netty-codec:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-handler-proxy:jar:4.1.58.Final:compile
[INFO] |  |  \- io.netty:netty-codec-socks:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-buffer:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-codec-http:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-codec-http2:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-transport-native-unix-common:jar:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.58.Final:compile
[INFO] |  +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.58.Final:compile
[INFO] |  \- io.projectreactor.netty:reactor-netty:jar:0.9.16.RELEASE:compile
[INFO] +- io.projectreactor:reactor-core:jar:3.4.34:compile
[INFO] |  \- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] +- com.azure:azure-core-management:jar:1.15.6:compile
[INFO] +- com.azure:azure-core:jar:1.54.1:compile
[INFO] |  +- com.azure:azure-xml:jar:1.1.0:compile
[INFO] |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.11.4:compile

• Java version: 11
• App Server/Environment: Maven
• Frameworks: Spring boot v2.3.8
  If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:
• verbose dependency tree (mvn dependency:tree -Dverbose)
• exception message, full stack trace, and any available logs

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[XiaofeiCao]

@wizzfizz94 Would you help provide HTTP request/response log?

See Enable Logging:

AzureResourceManager azure = AzureResourceManager
    .configure()
    .withLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS) // enable logging HTTP request body and headers
    .authenticate(credential, profile)
    .withDefaultSubscription();

Make sure you have slf4j implementation on the classpath, e.g. slf4j-simple:

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-simple</artifactId>
  <version>1.7.36</version> <!-- {x-version-update;org.slf4j:slf4j-simple;external_dependency} -->
  <scope>test</scope>
</dependency>

Don't forget to redact any sensitive information, thanks!

[wizzfizz94]

Thanks for the reply @XiaofeiCao

Request

{
    "az.sdk.message": "HTTP request",
    "method": "PUT",
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "tryCount": 1,
    "Date": "Tue, 18 Mar 2025 00:49:10 GMT",
    "x-ms-return-client-request-id": "true",
    "x-ms-client-request-id": "<x-ms-client-request-id>",
    "Accept": "application/json",
    "User-Agent": "azsdk-java-com.azure.resourcemanager.network/2.48.0 (11.0.26; Mac OS X; 14.7.4)",
    "redactedHeaders": "Authorization,Content-Type",
    "content-length": 191,
    "body": "{\"properties\":{\"peeringType\":\"AzurePrivatePeering\",\"peerASN\":<peer-asn>,\"primaryPeerAddressPrefix\":\"<primary-peer-address-prefix>\",\"secondaryPeerAddressPrefix\":\"<secondary-peer-address-prefix>\",\"vlanId\":<vlan-id>,\"microsoftPeeringConfig\":{}}}"
}

Response

{
    "az.sdk.message": "HTTP response",
    "statusCode": 201,
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "durationMs": 1882
}

Then it looks like the sdk createOrUpdate call triggers an additional get request following this with the following details.

Request

{
    "az.sdk.message": "HTTP request",
    "method": "GET",
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "tryCount": 1,
    "Date": "Tue, 18 Mar 2025 00:49:42 GMT",
    "x-ms-return-client-request-id": "true",
    "x-ms-client-request-id": "<x-ms-client-request-id>",
    "Accept": "application/json",
    "User-Agent": "azsdk-java-com.azure.resourcemanager.network/2.48.0 (11.0.26; Mac OS X; 14.7.4)",
    "redactedHeaders": "Authorization"
}

Response

{
    "az.sdk.message": "HTTP response",
    "statusCode": 404,
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "durationMs": 1788
}

subsequent requests to fetch all peerings for the associated cross connection also return an empty list.

[XiaofeiCao]

Thanks @wizzfizz94

Yes, express route private peering creation is a long-running operation, which follows a
submit create request -> poll until success -> get final result flow.

From the log, seems service backend accepted(201) the create request, though subsequently SDK failed to poll the provisioning state from the URL given by the initial response.

can confirm same details work via cURL

I'd like to know more details. Do you mean, when you initialize the PUT request via curl, subsequent requests to fetch all peerings for the associated cross connection will not return an empty list?

[wizzfizz94]

I'd like to know more details. Do you mean, when you initialize the PUT request via curl, subsequent requests to fetch all peerings for the associated cross connection will not return an empty list?

Correct, if i do the peering creation via curl PUT request a proceeding call to list peerings on the cross connection returns the created private peering. It seems that regardless of the 201 response from the PUT request with the sdk the peering is not created on the azure side.

[wizzfizz94]

Additionally, this call succeeds in deleting the peering

azure.networks().manager().serviceClient().getExpressRouteCrossConnectionPeerings()
.delete(
            resourceGroupName,
            serviceKey,
            "AzurePrivatePeering",
            com.azure.core.util.Context.NONE
);

Request

{
    "az.sdk.message": "HTTP request",
    "method": "DELETE",
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "tryCount": 1,
    "Date": "Tue, 18 Mar 2025 04:52:38 GMT",
    "x-ms-return-client-request-id": "true",
    "x-ms-client-request-id": "<x-ms-client-request-id>",
    "Accept": "application/json",
    "User-Agent": "azsdk-java-com.azure.resourcemanager.network/2.48.0 (11.0.26; Mac OS X; 14.7.4)",
    "redactedHeaders": "Authorization,Content-Type",
    "content-length": 0
}

Response

{
    "az.sdk.message": "HTTP response",
    "statusCode": 202,
    "url": "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/<resource-group-name>/providers/Microsoft.Network/expressRouteCrossConnections/<service-key>/peerings/AzurePrivatePeering?api-version=2024-05-01",
    "durationMs": 2280
}

But then throws this error

processRuntimeException: error msg: Long running operation failed., caused by ManagementException: Long running operation failed.

But can confirm that the deletion was successful on the azure side using the sdk call.

[XiaofeiCao]

processRuntimeException: error msg: Long running operation failed., caused by ManagementException: Long running operation failed.

Private peering deletion is also a long-running operation. The failure likely happened when retrieving resource status after submitting the DELETE request. Did you see any additional HTTP request/response following the DELETE response?