feat: Add clustering for loki.source.kubernetes_events

[petewall]

Brief description of Pull Request

This change adds a clustering option for loki.source.kubernetes_events that will distribute the work according to the list. If no namespaces are specificied, then only a single instance will run. This is great, because it means that it will be safe to run on Alloy instances with multiple replicas without resulting in duplication.

Pull Request Details

Issue(s) fixed by this Pull Request

Fixes #401

Notes to the Reviewer

PR Checklist

• Documentation added
• Tests updated
• Config converters updated

[github-actions]

💻 Deploy preview available (feat: Add clustering for loki.source.kubernetes_events):

• https://deploy-preview-alloy-6027-zb444pucvq-vp.a.run.app/docs/alloy/latest/

[kalleep]

Cool but I do think we will have at least one problem, that loki.source.kuberenetes also is affected by. It's that we use position file for tracking where we last read from.

The problem here and with that other component is that if a target moves to another instance we loose track of where we should start reading from and would start from beginning.

Not really a blocker itself but we should look into how we can solve this for both this one and loki.source.kubernetes.

[petewall]

A bit of testing:

1. Created a kind cluster
2. Deployed two alloys, both as 2-replica deployments, both using the same config:

          loki.source.kubernetes_events "cluster_events" {
            job_name   = "integrations/kubernetes/eventhandler"
            log_format = "logfmt"
            forward_to = [loki.write.loki.receiver]
            // These lines are only on the test alloy
            clustering {
              enabled = true
            }
          }

          loki.write "loki" {
            endpoint {
              url = "http://loki.loki.svc:3100/loki/api/v1/push"
              tenant_id = "1"
            }
            external_labels = {
              "source" = "control",
              "collector" = constants.hostname,
            }
          }

3. First alloy using latest published version, second alloy using my source-built image

Both control-group alloy replicas reported the same events (duplication)
only one of the test group alloy replicas send events (deduplication)

[petewall]

Cool but I do think we will have at least one problem, that loki.source.kuberenetes also is affected by. It's that we use position file for tracking where we last read from.

The problem here and with that other component is that if a target moves to another instance we loose track of where we should start reading from and would start from beginning.

Not really a blocker itself but we should look into how we can solve this for both this one and loki.source.kubernetes.

There's some good points here, but I'm not sure if it's big enough deal to hold this up. By default the TTL for kubernetes events is 1 hour, so any duplication would be bounded by that. And I feel the benefits (no more requirement for a singleton deployment) outweigh the concerns (potential short-term log duplication).

Is there an existing issue or a TODO for this, and we can try and fix that in another PR?

[kalleep]

Is there an existing issue or a TODO for this, and we can try and fix that in another PR?

Yeah there is a issue for loki.source.kubernetes as mentioned in slack. But I saw that you added a note about this and that is enough I think.