Skip to content

fix: status-reconciler track own config state for dropped deltas - #849

Open
Prucek wants to merge 1 commit into
kubernetes-sigs:mainfrom
Prucek:fix-statusreconciler-dropped-deltas
Open

fix: status-reconciler track own config state for dropped deltas#849
Prucek wants to merge 1 commit into
kubernetes-sigs:mainfrom
Prucek:fix-statusreconciler-dropped-deltas

Conversation

@Prucek

@Prucek Prucek commented Aug 14, 2026

Copy link
Copy Markdown
Member

Summary

  • config.Agent.Set() silently drops deltas when the subscriber channel is blocked for over a minute. When this happens, the agent's config advances but the subscriber never sees the notification, making those changes permanently invisible to status-reconciler.
  • Fix status-reconciler to diff against its own lastProcessed config rather than trusting the delta's Before field, so the next delivered delta captures all missed changes.
  • Add a warning log in the config agent when a delta delivery times out.

Fixes #848

Generated with the help of AI (Claude Code)

config.Agent.Set() silently drops deltas when the subscriber channel
is blocked for over a minute. When this happens, the agent's config
advances but the subscriber never receives the notification, making
those changes permanently invisible. This was observed in production
when status-reconciler spent hours processing a large delta and missed
new presubmit additions during that window.

Fix status-reconciler to diff against its own lastProcessed config
rather than trusting the delta's Before field, so the next delivered
delta captures all missed changes. Also log a warning in the config
agent when a delta is dropped.
@netlify

netlify Bot commented Aug 14, 2026

Copy link
Copy Markdown

Deploy Preview for k8s-prow ready!

Name Link
🔨 Latest commit e132ead
🔍 Latest deploy log https://app.netlify.com/projects/k8s-prow/deploys/6a7ece5bb00a460008185b8e
😎 Deploy Preview https://deploy-preview-849--k8s-prow.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Aug 14, 2026
@kubernetes-prow kubernetes-prow Bot added the area/status-reconciler Issues or PRs related to reconciling status when jobs change label Aug 14, 2026
@Prucek

Prucek commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

This seems like a flake: pull-prow-unit-test

@Prucek

Prucek commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

/test pull-prow-unit-test

@Prucek

Prucek commented Aug 14, 2026

Copy link
Copy Markdown
Member Author

/test pull-prow-integration

@kubernetes-prow

Copy link
Copy Markdown
Contributor

@kaovilai: changing LGTM is restricted to collaborators

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kaovilai, Prucek
Once this PR has been reviewed and has the lgtm label, please assign cblecker for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/status-reconciler Issues or PRs related to reconciling status when jobs change cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

status-reconciler can permanently miss config changes when delta delivery times out

2 participants