[Bug]: RoleAssignment stuck NotReady due to invalid observed status.atProvider.id format even when crossplane.io/external-name is correct after upgrade from provider 1.8 to 2.1

[callum-stakater]

Is there an existing issue for this?

• I have searched the existing issues

What I expected to happen:

If crossplane.io/external-name is corrected to the canonical AzureAD import ID format, the provider should:

 - Observe the external resource successfully using the corrected ID, and
 - Eventually reflect the canonical ID consistently in status.atProvider.id (or at least not continuously revert to an invalid/legacy format that breaks observe).

What actually happened

 - We corrected crossplane.io/external-name to the canonical ID format (e.g. /servicePrincipals/<spId>/appRoleAssignedTo/<assignmentId>).
 - However status.atProvider.id later shows (or reverts back to) the non-canonical / invalid ID format (missing the /servicePrincipals/.../appRoleAssignedTo/... prefix), and reconcile/observe fails again with the parsing error below.
 - Manually editing/removing status.atProvider.id is not durable; it is immediately reconciled back in incorrect format unless the roleassignment object is first forcefully deleted and recreated and reimported.

Affected Resource(s)

roleassignment.app.azuread.upbound.io/v1beta1

example MRs in failed state

apiVersion: app.azuread.upbound.io/v1beta1
kind: RoleAssignment
metadata:
  annotations:
    crossplane.io/composition-resource-name: foo
    crossplane.io/external-create-pending: "2025-12-16T11:02:08Z"
    crossplane.io/external-create-succeeded: "2025-12-16T11:02:08Z"
    crossplane.io/external-name: /servicePrincipals/fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignedTo/UzzR1ri5FkOz52DzzULUGuQzz2ia_xGzzImCpzzz
  creationTimestamp: "2025-12-16T11:02:07Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 2
  labels:
    crossplane.io/claim-name: foo
    crossplane.io/claim-namespace: foo
    crossplane.io/composite: foo
  name: foo
  ownerReferences:
  - apiVersion: iam.foo.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: XServicePrincipal
    name: baa
    uid: 413271f4-a1b5-489d-9174-50e8e12345678
  resourceVersion: "83832769"
  uid: 50aa26c6-5efe-433f-b994-b76e1fcb25ff
spec:
  deletionPolicy: Delete
  forProvider:
    appRoleId: 9f8d1818-ec88-888a-bc6c-a8e88b108808
    principalObjectId: d1d14b12-b1b8-4311-1ee1-10e1ce150151
    resourceObjectId: fzzz49zz-2450-4z15-z738-0z35z7f9zz0b
  initProvider: {}
  managementPolicies:
  - '*'
  providerConfigRef:
    name: azuread-application
status:
  atProvider:
    appRoleId: 9f9d1318-ec28-589a-bc6c-a8e05b103308
    id: fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignment/UkvR1ri5FkOe52DgzmULUGuQag2ia_xGjswImCpttnc
    principalDisplayName: foo
    principalObjectId: q6qs4q52-q9q8-4316-9aa7-60a0aa650b50
    principalType: Group
    resourceDisplayName: foo
    resourceObjectId: fzzz49zz-2450-4z15-z738-0z35z7f9zz0b
  conditions:
  - lastTransitionTime: "2025-12-16T11:02:09Z"
    reason: Available
    status: "True"
    type: Ready
  - lastTransitionTime: "2025-12-22T11:16:34Z"
    message: |-
      observe failed: failed to observe the resource: [{0 Parsing App Role Assignment ID parsing "fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignment/UkvR1ri5FkOe52DgzmULUGuQag2ia_xGjswImCpttnc": parsing the ServicePrincipalIdAppRoleAssignedTo ID: the number of segments didn't match

      Expected a ServicePrincipalIdAppRoleAssignedTo ID that matched (containing 4 segments):

      > /servicePrincipals/servicePrincipalId/appRoleAssignedTo/appRoleAssignmentId

      However this value was provided (which was parsed into 0 segments):

      > fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignment/UkvR1ri5FkOe52DgzmULUGuQag2ia_xGjswImCpttnc

      The following Segments are expected:

      * Segment 0 - this should be the literal value "servicePrincipals"
      * Segment 1 - this should be the user specified value for this servicePrincipalId [for example "servicePrincipalId"]
      * Segment 2 - this should be the literal value "appRoleAssignedTo"
      * Segment 3 - this should be the user specified value for this appRoleAssignmentId [for example "appRoleAssignmentId"]

      The following Segments were parsed:

      * Segment 0 - not found
      * Segment 1 - not found
      * Segment 2 - not found
      * Segment 3 - not found
       [{{} id}]}]
    observedGeneration: 2
    reason: ReconcileError
    status: "False"
    type: Synced
  - lastTransitionTime: "2025-12-16T11:02:08Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

Steps to Reproduce

happened after migration of provider-azuread from 1.8.x to 2.2.0, the ID in external-name was updated to reflect the new format but the status.atProvider.id value is not being updated during refresh

What happened?

when deleting an existing roleassignmement in this broken state and letting it recreate it hangs, removing the finalizer and letting it create in conflict with the existing assignment and then again patching the external-name with the same new format results in successful import and the status.atProvider.id DOES then get populated with the correct NEW ID format

Relevant Error Output Snippet

    message: |-
      observe failed: failed to observe the resource: [{0 Parsing App Role Assignment ID parsing "fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignment/UkvR1ri5FkOe52DgzmULUGuQag2ia_xGjswImCpttnc": parsing the ServicePrincipalIdAppRoleAssignedTo ID: the number of segments didn't match

      Expected a ServicePrincipalIdAppRoleAssignedTo ID that matched (containing 4 segments):

      > /servicePrincipals/servicePrincipalId/appRoleAssignedTo/appRoleAssignmentId

      However this value was provided (which was parsed into 0 segments):

      > fzzz49zz-2450-4z15-z738-0z35z7f9zz0b/appRoleAssignment/UkvR1ri5FkOe52DgzmULUGuQag2ia_xGjswImCpttnc

      The following Segments are expected:

      * Segment 0 - this should be the literal value "servicePrincipals"
      * Segment 1 - this should be the user specified value for this servicePrincipalId [for example "servicePrincipalId"]
      * Segment 2 - this should be the literal value "appRoleAssignedTo"
      * Segment 3 - this should be the user specified value for this appRoleAssignmentId [for example "appRoleAssignmentId"]

      The following Segments were parsed:

      * Segment 0 - not found
      * Segment 1 - not found
      * Segment 2 - not found
      * Segment 3 - not found
       [{{} id}]}]

Crossplane Version

1.20

Provider Version

2.2.0

Kubernetes Version

No response

Kubernetes Distribution

No response

Additional Info

No response

[jonasz-lasut]

Hi @callum-stakater thank you for opening the issue!

I was able to reproduce the issue in both Crossplane 1.20 and in Crossplane 2.1.3.
There are two parts to the answer, first will allow you to update existing resources, second will be about the bug behavior.

Updating existing resources without recreation:
You can update the existing resources to use the new external-name/id by following the steps listed below:

• Update the crossplane.io/external-name: annotation
• Scale provider-azuread to 0 replicas using DeploymentRuntimeConfig

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-azuread
spec:
  package: xpkg.upbound.io/upbound/provider-azuread:v2.2.0
  runtimeConfigRef:
    name: upbound-provider-azuread
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: upbound-provider-azuread
spec:
  deploymentTemplate:
    spec:
      replicas: 0
      selector: {}
      template: {}

• Scale-back the provider

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: upbound-provider-azuread
spec:
  package: xpkg.upbound.io/upbound/provider-azuread:v2.2.0
  runtimeConfigRef:
    name: upbound-provider-azuread
---
apiVersion: pkg.crossplane.io/v1beta1
kind: DeploymentRuntimeConfig
metadata:
  name: upbound-provider-azuread
spec:
  deploymentTemplate:
    spec:
      replicas: 1
      selector: {}
      template: {}

• This leads to status.atProvider.id getting refreshed and resource being queued for reconciliation (default 10m)

judasz@fedora:~/code/provider-upjet-azuread$ k logs -n crossplane-system -l pkg.crossplane.io/provider=upbound-provider-azuread
2026-01-02T10:20:39Z    DEBUG   provider-azuread        Reconciling     {"controller": "managed/serviceprincipals.azuread.upbound.io/v1beta1, kind=principal", "request": {"name":"id-not-updating"}}
2026-01-02T10:20:39Z    DEBUG   provider-azuread        Connecting to the service provider      {"uid": "c9d4efab-7794-45eb-8166-9eba9366f968", "name": "id-not-updating", "namespace": "", "gvk": "serviceprincipals.azuread.upbound.io/v1beta1, Kind=Principal"}
2026-01-02T10:20:39Z    DEBUG   provider-azuread        Connecting to the service provider      {"uid": "277deb24-4341-483c-b1a1-05483a8adce2", "name": "id-not-updating", "namespace": "", "gvk": "applications.azuread.upbound.io/v1beta1, Kind=Application"}
2026-01-02T10:20:39Z    DEBUG   provider-azuread        Reconciling     {"controller": "providerconfig/providerconfig.azuread.upbound.io", "request": {"name":"default"}}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        Instance state not found in cache, reconstructing...    {"uid": "277deb24-4341-483c-b1a1-05483a8adce2", "name": "id-not-updating", "namespace": "", "gvk": "applications.azuread.upbound.io/v1beta1, Kind=Application"}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        Observing the external resource {"uid": "277deb24-4341-483c-b1a1-05483a8adce2", "name": "id-not-updating", "namespace": "", "gvk": "applications.azuread.upbound.io/v1beta1, Kind=Application"}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        Instance state not found in cache, reconstructing...    {"uid": "c9d4efab-7794-45eb-8166-9eba9366f968", "name": "id-not-updating", "namespace": "", "gvk": "serviceprincipals.azuread.upbound.io/v1beta1, Kind=Principal"}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        Observing the external resource {"uid": "c9d4efab-7794-45eb-8166-9eba9366f968", "name": "id-not-updating", "namespace": "", "gvk": "serviceprincipals.azuread.upbound.io/v1beta1, Kind=Principal"}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        External resource is up to date {"controller": "managed/applications.azuread.upbound.io/v1beta1, kind=application", "request": {"name":"id-not-updating"}, "uid": "277deb24-4341-483c-b1a1-05483a8adce2", "version": "33700", "external-name": "/applications/7053f512-b9be-474f-b351-348c059d18b3", "requeue-after": "2026-01-02T10:30:17Z"}
2026-01-02T10:20:40Z    DEBUG   provider-azuread        External resource is up to date {"controller": "managed/serviceprincipals.azuread.upbound.io/v1beta1, kind=principal", "request": {"name":"id-not-updating"}, "uid": "c9d4efab-7794-45eb-8166-9eba9366f968", "version": "33868", "external-name": "/servicePrincipals/6ff1188f-5dd9-46d8-8971-64b53fd89e37", "requeue-after": "2026-01-02T10:31:06Z"}

Provider using old status.atProvider.id value instead of value based on external-name when the id structure is changed due to terraform provider changes:
The behavior seems to be a bug in the underluing runtime and not directly in provider-azuread as I was able to replicate it with provider-azure as well. The status.atProvider.id is not getting updated when the id structure is changed in the underlying terraform provider resource. I'll prepare an example that can be easily reproduced based on what you have prepared and open an issue upstream.

Please let me know if the resolution that I've proposed is fine for you and if the update of resources without recreation works as expected and I'll close the issue after

[callum-stakater]

Ah nice, I didn't think about offlining the provider before stripping out the status, sounds promising, will try and report back. Thanks

[jonasz-lasut]

Hi @callum-stakater I've run some more tests and it looks like you don't even need to strip out the status, scaling the provider to zero and back is enough to update the status.atProvider.id and then you need to wait for reconciliation, which is 10 minutes by default in provider-azuread.
I've updated the original comment

[callum-stakater]

yeah was about to report the same, it was sufficient to scale the provider down and backup for it to reconcile the ids

[jonasz-lasut]

@callum-stakater FYI one of the maintainers have opened a PR at Upjet and it looks like it leads to correct behavior of the provider - crossplane/upjet#564

After it's merged I'll update the upjet version in provider-upjet-azuread and provider-upjet-azure and that should lead to the provider automatically picking up the new external-name without a need to scale back/up. I hope to have this fix as part of the next release