Skip to content

[receiver/k8sobjectsreceiver] k8sobject receiver support leader election #39054

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Frapschen
Copy link
Contributor

Description

As #38426 move the k8sleaderelector from development to alpha, we can use it in k8sobjectsreceiver.

@Frapschen
Copy link
Contributor Author

I have test the code in my enviroment. I use this config:

exporters:
  debug: {}
extensions:
  k8s_leader_elector/k8sobjects:
    auth_type: kubeConfig
    context: cluster-bm-admin@cluster-bm
    lease_name: local-test-k8sobjects
    lease_namespace: default
processors:
  batch: {}
receivers:
  k8sobjects:
    auth_type: kubeConfig
    context: cluster-bm-admin@cluster-bm
    k8s_leader_elector: k8s_leader_elector/k8sobjects
    objects:
      - name: events
        mode: pull
        group: events.k8s.io/v1
        interval: 10s
service:
  telemetry:
    logs:
      level: info
    metrics:
      address: 0.0.0.0:1233
  extensions:
    - k8s_leader_elector/k8sobjects
  pipelines:
    logs:
      exporters:
#        - file
        - debug
      processors:
        - batch
      receivers:
        - k8sobjects

test case 1: start two collector instances with leader electon
image
we can see only the first collector successfully acquired lease and start pull k8s event data

test case 1: The lease resource was accidentally deleted
image
The first collector print error logs but stil get lease and continue pull event from k8s

test case 1: shutdown the first and wait 3s to restart
image
The second collector successfully acquired the lease 20 seconds after the first collector shut down.

@Frapschen Frapschen force-pushed the k8s0bject-leader-election branch from 4bb004b to 81d5190 Compare March 31, 2025 11:05
@Frapschen Frapschen force-pushed the k8s0bject-leader-election branch from 81d5190 to 00284d7 Compare April 1, 2025 06:22
},
func() {
kr.setting.Logger.Info("Object Receiver stopped as leader lose")
_ = kr.Shutdown(context.Background())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you might like to log that error.

}

kr.setting.Logger.Info("Trying to become the leader")
elector := k8sLeaderElector.(k8sleaderelector.LeaderElection)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please check the casting to avoid panics.

@atoulme
Copy link
Contributor

atoulme commented Apr 5, 2025

Please resolve conflicts and add tests. It would be ideal to have a unit test that covers an election with receivers joining and leaving.

@atoulme atoulme marked this pull request as draft April 5, 2025 05:58
@Frapschen Frapschen force-pushed the k8s0bject-leader-election branch from 1bed805 to c7a8865 Compare April 6, 2025 16:01
@Frapschen
Copy link
Contributor Author

@atoulme A normal unit test can‘t achieve your suggestion. I will try to add a new e2e test to address this.

Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Apr 24, 2025
@Frapschen Frapschen removed the Stale label Apr 24, 2025
@rakesh-garimella
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants