Reconciliation generates watch events on Objects with no changes

[bobh66]

What happened?

We noticed that there are a large number of no-change update watch events happening in deployments that contain a large number of Object resources.

We are seeing two watch events for every Object in every reconciliation cycle, even when there are no changes
being applied to the objects. The first watch event is triggered by a status update from the provider:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"3b346756-ad1f-42f5-ae0f-62d273f40402","stage":"ResponseComplete","requestURI":"/apis/kubernetes.crossplane.io/v1alpha2/objects/bobncom-ngsjm-8f6f35d00355/status","verb":"update","user":{"username":"system:serviceaccount:crossplane-system:provider-kubernetes","uid":"b81295bd-4f57-4fae-9c8e-a280ae7145ae","groups":["system:serviceaccounts","system:serviceaccounts:crossplane-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=3a085658-fd97-41df-8c4b-e0a1ee20edda"],"authentication.kubernetes.io/node-name":["ip-10-254-0-45.ec2.internal"],"authentication.kubernetes.io/node-uid":["8a56b5dc-c208-477f-8ae4-422bbfd832e4"],"authentication.kubernetes.io/pod-name":["provider-kubernetes-75f7657d8d-pdpcr"],"authentication.kubernetes.io/pod-uid":["b715f9ea-5c3a-4b7a-9d3f-a23003a7bf26"]}},"sourceIPs":["10.254.14.242"],"userAgent":"crossplane-kubernetes-provider/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"objects","name":"bobncom-ngsjm-8f6f35d00355","uid":"b6b2664a-925d-495c-934f-9daa7930e180","apiGroup":"kubernetes.crossplane.io","apiVersion":"v1alpha2","resourceVersion":"4345770","subresource":"status"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2026-01-12T23:43:02.299010Z","stageTimestamp":"2026-01-12T23:43:02.324430Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crossplane:provider:provider-kubernetes-4da041ccd79c:system\" of ClusterRole \"crossplane:provider:provider-kubernetes-4da041ccd79c:system\" to ServiceAccount \"provider-kubernetes/crossplane-system\""}}

This status update triggers a large number of updates to the composite, on the order of 300 log messages:

2026-01-13T01:39:31Z	DEBUG	crossplane	Mapping composite resource because composed resource changed	{"controller": "defined/compositeresourcedefinition.apiextensions.crossplane.io", "namespace": "", "name": "bobncom-ngsjm-dd9bef04aa26", "cdGVK": "kubernetes.crossplane.io/v1alpha1, Kind=Object", "cdName": "bobncom-ngsjm-8f6f35d00355"}

and the second event occurs a few seconds later in response to the status update:

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"372aa3e6-761f-4f5e-abe0-7ea31ea7d09b","stage":"ResponseComplete","requestURI":"/apis/kubernetes.crossplane.io/v1alpha1/objects/bobncom-ngsjm-8f6f35d00355?fieldManager=apiextensions.crossplane.io%2Fcomposed%2F52fe4d13cf623b18c3bb78b1db3201ab61f3c8c2b3891864e5a9a91a371f5a72\u0026force=true","verb":"patch","user":{"username":"system:serviceaccount:crossplane-system:crossplane","uid":"4ac000ef-5307-4b24-acb9-15cd33d24535","groups":["system:serviceaccounts","system:serviceaccounts:crossplane-system","system:authenticated"],"extra":{"authentication.kubernetes.io/credential-id":["JTI=59f1b291-552d-437e-b526-0af05ea32b07"],"authentication.kubernetes.io/node-name":["ip-10-254-5-117.ec2.internal"],"authentication.kubernetes.io/node-uid":["c66fda7a-55a8-43d3-a6b8-18788e5a037f"],"authentication.kubernetes.io/pod-name":["crossplane-66cdfd8f66-4ld89"],"authentication.kubernetes.io/pod-uid":["4fdc0811-26ee-4b0f-96d4-e61d07224cb3"]}},"sourceIPs":["10.254.14.190"],"userAgent":"crossplane/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"objects","name":"bobncom-ngsjm-8f6f35d00355","apiGroup":"kubernetes.crossplane.io","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2026-01-12T23:43:02.137843Z","stageTimestamp":"2026-01-12T23:43:02.183072Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"crossplane\" of ClusterRole \"crossplane\" to ServiceAccount \"crossplane/crossplane-system\"","k8s.io/deprecated":"true"}}

This cycling of updates only happens when the Object is deployed via a Composition, so the two reconcilers are continuously making no-change updates to the same resource and triggering watch events.

The metadata.resourceVersion changes on every watch event but no other fields are changed.

How can we reproduce it?

Deploy Objects using a composition, start a watch on the Object and observe the events on every reconciliation cycle but no changes to the Object.

What environment did it happen in?

Crossplane version: 2.1.3
EKS 1.33

[bobh66]

I think the status update may be coming from the managed reconciler in crossplane-runtime - should it be using SSA for status updates?

[jbw976]

@bobh66 do you think root cause of this issue is crossplane/crossplane-runtime#902?

• Managed Resources are generating watch events on every poll interval due to missing observedGeneration in the Synced condition crossplane/crossplane-runtime#902

[bobh66]

@jbw976 originally I thought it might, but I'm not seeing the Synced condition lastTransitionTime changing on the Object so I think it's a different problem. The only field I see changing in the Object is the metadata.resourceVersion which means the resource is being stored in etcd.

Another interesting thing is that it only happens when the Object is created by a composition - with or without realtime compositions. Something in the composite reconciler is triggering reconciliation in the provider, which then triggers the composite either immediately when realtime compositions is enabled, or eventually when the polling interval expires.

If I create an Object manually without a composite the problem doesn't happen.I even turned on changelogs to look for update events but that only helps when the managed resource Update() method is called, which it isn't in this case (or in the crossplane-runtime scenario either).

The audit logs show the entries above but no indication of what the API server thinks is changing.

[bobh66]

I think there are two questions that need to be answered:

1 - why does the no-change Object status update by the managed reconciler generate a watch event for the composite reconciler? Meaning why does it get stored in etcd and cause a resourceVersion change which creates the watch event?

2 - why does the SSA of the composed Object from the composite reconciler - which should also be a no-change update - cause a reconciliation of the Object by the provider?

If either of these events doesn't happen then we don't have a reconcilation loop, but neither of them should happen since they are both no-change updates.

[bobh66]

Refining those two questions a little more:
1 - should the managed reconciler be using SSA - client.Status().Apply() - instead of client.Status().Update() for the status updates?

2 - should the watches created by the composite resource reconciler be using an EventFilter similar to what has been implemented in the provider reconcilers to block unnecessary reconciles?

[bobh66]

This is easily reproducible in a kind cluster with a trivial composition that creates an Object. Using the latest crossplane and the latest provider-kubernetes, the provider will generate a watch event every pollInterval for what appears to be no reason, which triggers the composite reconciler for no reason.

This isn't a problem for one Object, but when you have thousands of Objects in a cluster and multiple/many Objects in a composite, the unnecessary reconciliations start to add up very quickly.

[bobh66]

I disabled realtime compositions and the problem remains so it's not related to that feature.

[haarchri]

@bobh66 does this happen also when you disable server-side-apply in provider-kubernetes, btw using a version before server-side-apply was enabled by default ?

[bobh66]

@haarchri I believe so but I'll run a test to verify.

[bobh66]

@haarchri yes it happens when SSA is disabled in the provider.

[bobh66]

@haarchri it appears to happen only for v1alpha1 Objects - if I use v1alpha2 Objects then there are no watch events generated on reconciliation. Maybe the conversion webhook is causing the problem?

[ytsarev]

I was just hit by this issue the Object is activating the circtuit breaker

  conditions:
  - lastTransitionTime: "2026-03-12T22:14:06Z"
    message: Too many watch events from Object/redis-fallback-e2e-app (default). Allowing
      events periodically.
    observedGeneration: 5
    reason: WatchCircuitOpen
    status: "False"
    type: Responsive

At the same time no activity is happening around Object itself - 1 stale event in the stream

k describe objects.kubernetes.m.crossplane.io redis-fallback-e2e-app
Name:         redis-fallback-e2e-app
Namespace:    default
Labels:       crossplane.io/composite=e2e-cache
Annotations:  crossplane.io/composition-resource-name: fallback-deployment
              crossplane.io/external-create-pending: 2026-03-12T22:14:06Z
              crossplane.io/external-create-succeeded: 2026-03-12T22:14:06Z
              crossplane.io/external-name: redis-fallback-e2e-app
API Version:  kubernetes.m.crossplane.io/v1alpha1
Kind:         Object
...
Events:
  Type    Reason                   Age    From                                       Message
  ----    ------                   ----   ----                                       -------
  Normal  CreatedExternalResource  5m26s  managed/object.kubernetes.m.crossplane.io  Successfully requested creation of external resource