Fix: Schedule scanner false-positives for inactive NS - #11703
Open
davidporter-id-au wants to merge 2 commits into
Open
Fix: Schedule scanner false-positives for inactive NS #11703davidporter-id-au wants to merge 2 commits into
davidporter-id-au wants to merge 2 commits into
Conversation
…e invariant Two independent sources of false positives in the schedule-invariants scanner, both found while chasing a "stuck" V2 schedule that turned out to be healthy - the dump under investigation was a standby replica 3h24m behind its active cluster. 1. No active-cluster gate. ListAllNamespaces filtered only on NAMESPACE_STATE_DELETED, so every cluster scanned every namespace. Each invariant here is evaluated from visibility records, and a standby's records track its replica: a schedule ticking normally on the active side looks arbitrarily overdue on a standby, whose ScheduleNextActionTime is frozen at whatever it was when replication last landed. Every passive cluster re-reported the whole namespace as anomalous. The ingredient was already wired in and dropped on the floor - Activities.currentClusterName was declared, passed to the constructor, assigned, and never read. The sibling build-ids scavenger does this correctly (scavenger.go:172), as does pernamespaceworker.go:383. The function's own doc comment already claimed it returned "every namespace active in the current cluster"; now it does. TestListAllNamespaces_FiltersInactiveAndDeleted was pinning the bug: it built a globalNS explicitly commented "inactive in this cluster" and then asserted that it WAS returned. The assertion now matches the test's name. 2. The confirmation step never re-checked the invariant. scheduleIsExpectedNotToFire called DescribeSchedule and then looked only at two unrelated exemptions (paused; BUFFER_ONE/BUFFER_ALL with a running workflow) - never at whether the next action time was still overdue in authoritative state. A candidate is now cleared when Info.FutureActionTimes shows no overdue action, which makes the check self-correcting against any stale index entry - replication lag, visibility indexing lag, or a failover - instead of depending on enumerating every benign explanation. Deliberately not added as an exemption: SKIP with a running workflow. A SKIP schedule whose action overruns the interval does drop that action, but the Generator still ticks and advances FutureActionTimes, so the re-check clears it without an exemption that would also mask a genuinely stalled SKIP schedule. The re-check suppresses candidates, so it emits schedule_invariants_scanner_overdue_next_action_time_stale_candidate rather than going silently blind: a sustained nonzero rate means visibility is lagging, and a rate that swallows every candidate means the re-check itself has broken. Note the count-only scanners (stuck_open, unknown_state) have no confirmation step at all, so fix 1 is their only protection. All new and changed tests were verified to fail with the fixes reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
davidporter-id-au
force-pushed
the
scanner-active-cluster-and-invariant-recheck
branch
from
August 21, 2026 05:32
11c8c67 to
47da2a6
Compare
davidporter-id-au
changed the title
davidporter-id-au
marked this pull request as ready for review
August 21, 2026 05:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This fixes the Schedule invariant scanner's false-positives coming from replication. Due to carelessness it was firing on the passive side because I forgot to filter this out, and for a while during post-replication disconnection, the task processing will cease. Also adds a small check for Described schedules to filter out visibility drift.
How
Risks: