Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[processor/k8sattributes] Wait for ReplicaSet informer before starting pod informer #37138

Open
wants to merge 26 commits into
base: main
Choose a base branch
from

Conversation

bacherfl
Copy link
Contributor

Description

This PR is an alternative approach to the previous fix made for #37056, which did not fully solve the issue of the deployment name not being added to a pod after the initial informer sync

Link to tracking issue

Fixes #37056

Testing

Re-enabled the flaky E2E test

@github-actions github-actions bot added the processor/k8sattributes k8s Attributes processor label Jan 10, 2025
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
Signed-off-by: Florian Bacher <[email protected]>
@bacherfl
Copy link
Contributor Author

after running the tests a number of times, it looks like the e2e tests are passing consistently now, so this should be ready for review

@bacherfl bacherfl marked this pull request as ready for review January 14, 2025 08:29
@bacherfl bacherfl requested a review from a team as a code owner January 14, 2025 08:29
Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some suggestions based on the following logic:

Step 1: wait for all Informers to get synced before unblocking the Pod's informer start
Step 2. Based on the waitForMetadata setting block on Pod's informer sync which is already dependent to the sync of the rest of the informers.

In this, the Pod's informer sync comes as a task chord which is the only one we need to block on from the main flow since it's already blocked on the rest "internally".

processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
@bacherfl
Copy link
Contributor Author

I left some suggestions based on the following logic:

Step 1: wait for all Informers to get synced before unblocking the Pod's informer start Step 2. Based on the waitForMetadata setting block on Pod's informer sync which is already dependent to the sync of the rest of the informers.

In this, the Pod's informer sync comes as a task chord which is the only one we need to block on from the main flow since it's already blocked on the rest "internally".

Thanks for the review! Those suggestions make sense, I will look into updating the PR accordingly now

Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank's!

processor/k8sattributesprocessor/internal/kube/client.go Outdated Show resolved Hide resolved
@ChrsMark
Copy link
Member

@fatsheep9146 @TylerHelmuth @dmitryax could you take a look as well?

if err != nil {
return err
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not add podInformer's HasSynced into the slice synced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
processor/k8sattributes k8s Attributes processor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Prometheus metrics missing k8s_deployment_name attribute for short period after agent restart
4 participants