Implement Kubernetes Event API v1

[adri1197]

Transition the Flux ecosystem from legacy core/v1 events to the modern events.k8s.io/v1 API. The following changes have to follow a backward compatibility approach where legacy fields may be bridged to the new schema (e.g., keep InvolvedObject regardless of being renamed, add a new RelatedObject field) to ensure no breaking changes for users relying on current event logs.

Action Items

• Add a new AnnotatedEventRecorder interface upstream in client-go/tools/events - events: add AnnotatedEventRecorder interface into internal/events kubernetes/kubernetes#138103
• Controller-Runtime - ⚠️ Add AnnotatedEventRecorderProvider interface kubernetes-sigs/controller-runtime#3509
• Update fluxcd/pkg to use events.k8s.io/v1 - runtime: Migrate to new Kubernentes event API pkg#1144
• Update all controllers using the changes done in runtime/events from fluxcd/pkg. Use new field relatedObject to include object's source metadata (as InvolvedObject does) and Action as mandatory field.
  • Mandatory controllers
    • Notification-Controller - Migrate to new Kubernetes events API notification-controller#1267
      • Adapt the event receiver to handle the events.k8s.io/v1 schema.
      • Update spec/v1beta3/events.md to document new fields.
    • Helm-Controller Migrate event recording to Flux events v1 API helm-controller#1510
    • Kustomize-Controller Migrate event recorder to fluxcd/pkg/runtime/events.Recorder kustomize-controller#1621
    • Source-Controller Migrate event recorder to event/v1 source-controller#2011
  • Optional controllers
    • Source-Watcher Adopt k8s event/v1 API with semantic actions source-watcher#343
    • Image-Automation-Controller Migrate event recorder to fluxcd/pkg/runtime/events.Recorder image-automation-controller#1038
    • Image-Reflector-Controller Adopt pkg/runtime/events.EventRecorder in reconcilers image-reflector-controller#889

Reference to the new API spec and changes

• API Deprecation Guide ( events.k8s.io/v1beta1 deprecated since k8s 1.25)
• KEP-383: New Event API GA Graduation

[adri1197]

Hi @matheuscscp,

You were right that the k8s.io/client-go/tools/events interface doesn't support setting annotations, so we should stick with a simple Eventf. I'm looking into whether it would be feasible to propose bringing the AnnotationEventf function back upstream in Kubernetes.

In any case, I don't think this should be a major issue, since we can still send the annotation to notification-controller using github.com/fluxcd/pkg/apis/event/v1. I'll need to rework some of the controller test cases so they don't rely on annotations to filter Kubernetes events.

[matheuscscp]

Does this mean we're losing support for annotations in k8s events?

[adri1197]

Does this mean we're losing support for annotations in k8s events?

Yes, seems so. I was thinking on contributing to k8s/client-go/tools and bring back AnnotatedEventf(...). Do you think it may be feasible? As any object, the interface allows to add annotations, however it does not seem to be exposed in the new Eventf(...) function.

[stefanprodan]

Without annotations all our unit tests around events will break in all controllers. Also this would be a major breaking change for those that collect Flux events in Loki and other systems...

[adri1197]

Without annotations all our unit tests around events will break in all controllers. Also this would be a major breaking change for those that collect Flux events in Loki and other systems...

Yes, I have been struggling with that... Let's bring it up to k8s then to be able to set annotations.

[stefanprodan]

@adri1197 can we use the old functions with the new structure?

[adri1197]

@adri1197 can we use the old functions with the new structure?

I still need to test it, but I believe so — it should just be a matter of using the old Recorder.

By the way, I’ve opened a PR in Kubernetes to add AnnotatedEventf(...) back to the EventRecorder interface: kubernetes/kubernetes#138103. Not sure yet if it will be accepted, but at least the proposal is there.

[stefanprodan]

@adri1197 Kubernetes v1.36 is going to be released at the end of April, so unless your upstream PR gets merged this week, we'll miss the window for Flux 2.9

[adri1197]

@adri1197 Kubernetes v1.36 is going to be released at the end of April, so unless your upstream PR gets merged this week, we'll miss the window for Flux 2.9

Yeah, it’s starting to look like getting this upstream isn’t going to happen, at least not any time soon.

I'm going to start reverting back to the old recorder, while still keeping the new events API in place where it makes sense. That should let us keep the current behavior (including annotations) without blocking on the Kubernetes side.

[adri1197]

@adri1197 Kubernetes v1.36 is going to be released at the end of April, so unless your upstream PR gets merged this week, we'll miss the window for Flux 2.9

Yeah, it’s starting to look like getting this upstream isn’t going to happen, at least not any time soon.

I'm going to start reverting back to the old recorder, while still keeping the new events API in place where it makes sense. That should let us keep the current behavior (including annotations) without blocking on the Kubernetes side.

Looks like it's not going to be possible with the old recorder. As it expects core/v1, the main functions Evenf(...) and AnnotatedEventf(...), they may not have the same arguments and therefore, it's not going to be able to implement k8s.io/client-go/tools/record.

Unless we would like to switch to k8s.io/api/events/v1, without leveraging none of the new fields added in the new API version.

[matheuscscp]

I'd wait for Kubernetes upstream to merge your PR

[adri1197]

Quick Update: after discussing in the SIG Instrumentation bi-weekly call, the consensus is to include AnnotatedEventf in the new events API for its potential correlation benefits.

Let's see how the review unfolds.