bug: api-syncagent generates thousands of http 404 request #126
Description
Describe the bug
We might have discovered a bug somewhere in api-syncagent, multicluster-runtime or kcp (not sure tbh): We are experiencing that the api-syncagent is receiving reconcile requests for objects in virtual workspaces that target the wrong cluster ID which ends up in 404 error. The object in the backend cluster as well as the state secret there contain a different (but the correct) cluster ID. Meanwhile the syncagent fires thousands of request that point to the wrong resource. Currently, of 700k requests, we have around 80% 404, on average 4k per minute - all by api-syncagent.
For example, the agent calls the following API (for which kcp returns a 404):
/services/apiexport/19kwudfbgwwhwnm8/mygroup.example.com/clusters/2446atsin4rofd86/apis/mygroup.example.com/v1alpha1/myexamples/my-resource-name
But the object is actually in workspace 8cr8c9tcvo5vamt5. This also specified in the backend cluster label:
syncagent.kcp.io/remote-object-cluster=8cr8c9tcvo5vamt5
The same goes for the secret storing the object state.
Steps To Reproduce
We don't know how to reproduce this. Any hint how to debug this would be appreciated.
Version information:
- kcp:
v0.28.3 - api-syncagent:
v0.4.2
PublishedResource:
apiVersion: syncagent.kcp.io/v1alpha1
kind: PublishedResource
metadata:
name: myexample.v1alpha1.mygroup.example.com
spec:
naming:
name: '{{ $ns := default .ClusterName .Object.metadata.namespace }}{{ $name :=
.Object.metadata.name }}{{ $combined := printf "%s-%s" $ns $name }}{{ if gt
(len $combined) 63 }}{{ $truncated := substr 0 36 $name }}{{ $hash := sha256sum
$name | substr 0 5 }}{{ printf "%s-%s-%s" $ns $truncated $hash }}{{ else }}{{
$combined }}{{ end }}'
namespace: '{{ printf "ws-%s" .ClusterName }}'
resource:
apiGroup: mygroup.example.com
kind: MyExample
version: v1alpha1
status:
resourceSchemaName: v9b134268.myexamples.mygroup.example.com
(replaced some strings due to sensitive information)
Expected Behaviour
The api-syncagent should send requests for objects in the right cluster path.
Additional Context
No response