Skip to content

marking consolidatable log messages are duplicated due to read-after-write inconsistency #2002

Open
@jonathan-innis

Description

@jonathan-innis

Description

Observed Behavior:

{"level":"DEBUG","time":"2025-02-17T17:23:51.179Z","logger":"controller","caller":"disruption/controller.go:99","message":"marking consolidatable","commit":"a79abac","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-pwzrz"},"namespace":"","name":"default-pwzrz","reconcileID":"05fd3c4d-cad7-4979-862f-de6cee15988e","Node":{"name":"ip-192-168-106-221.us-west-2.compute.internal"}}
{"level":"DEBUG","time":"2025-02-17T17:23:51.372Z","logger":"controller","caller":"disruption/controller.go:99","message":"marking consolidatable","commit":"a79abac","controller":"nodeclaim.disruption","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-pwzrz"},"namespace":"","name":"default-pwzrz","reconcileID":"957af1c8-0966-47ea-be7e-8ae2697ca4df","Node":{"name":"ip-192-168-106-221.us-west-2.compute.internal"}}

Karpenter will double-log that a status condition has transitioned for Consolidatable (the same probably applies to Drifted) because there is no read-after-write consistency so Karpenter needs enough time after it has mutated the object to actually see the cache update -- one way to resolve this is to do an event filter on status updates -- the other way to accomplish this is by adding an artificial sleep into the Reconcile to give the cache time to catch-up like here

Expected Behavior:

Karpenter should not produce the same log twice in a row.

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

good first issueDenotes an issue ready for a new contributor, according to the "help wanted" guidelines.help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions