Server rejected event errors when leader election is enabled

[MatousJobanek]

When the leader election is enabled:

restConfig := ctrl.GetConfigOrDie()
...
mgr, err := kcp.NewClusterAwareManager(cfg, ctrl.Options{
	Scheme:                 scheme,
	MetricsBindAddress:     metricsAddr,
	Port:                   9443,
	HealthProbeBindAddress: probeAddr,
	LeaderElection:         true,
	LeaderElectionConfig:   restConfig,
	LeaderElectionID:       "68a0532d.my.domain",
})

(NOTE: I specify the original rest config so it doesn't use the VirtualWorkspace URL, otherwise, it would fail because the client used in leader election doesn't use the right round tripper.)

then the leader election code tries to create an event in the VirtualWorkspace URL, but it fails because it's using a wrong path (does not contain the expected /clusters/* section)

I0707 11:39:21.287347       1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 0 milliseconds
E0707 11:39:21.287479       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62aeb2", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"bc0f0a11-21c8-4a6a-b632-6d5abe3a7532", APIVersion:"v1", ResourceVersion:"27429", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)
I0707 11:39:21.289195       1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 1 milliseconds
E0707 11:39:21.289449       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62d112", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"b5d779ea-914b-41e5-aa3a-71ff06ee5349", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"27430", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)

Instead of using the VirtualWorkspace URL, it should use the URL of the actual workspace that was provided as part of LeaderElectionConfig

[kcp-ci-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale