Auto-Recover Stuck Queue Builds And Fix Queue Status API

[frostebite]

Summary

• fix queueStatus cross-origin access for the docs admin UI
• filter queue-status build data by Docker repo version
• automatically recover maxed-out failed builds for the latest repo version, with safety caps so broken builds cannot loop forever
• fix retryBuild to use the job's repo version instead of the latest

Problem

The docs site calls queueStatus cross-origin from game.ci. Without CORS headers the browser fails the request with Failed to fetch, which breaks Admin Queue Management.

Separately, retry-exhausted failed builds required a manual resetFailedBuilds call before the queue could retry them again. Because the scheduler prioritises failed jobs before new created editor jobs, that left newer editor versions stuck behind stale failures and required admin intervention. We saw this on repo version 3.2.2 (Total 7108, Published 7099, In progress 0, ~9 failed) where created jobs for newer editors were showing the builder-head icon on the docs site indefinitely.

Changes

• add CORS + OPTIONS handling to queueStatus
• support repo-version-scoped build queries in queueStatus (jobs remain global, builds are filtered)
• add repo-version-scoped lookup for maxed-out failed builds
• retryBuild: dispatch with the build's own jobData.repoVersionInfo rather than RepoVersionInfo.getLatest(), so retrying an older repo version no longer mislabels the dispatch as the current latest
• during the scheduled queue pass, automatically recover maxed-out failed builds for the latest repo version:
  • if the image already exists on Docker Hub, mark it published (same logic as the existing healFailedBuildsAlreadyOnDockerHub heuristic, but applied to the maxed-out subset)
  • otherwise reset its failure count so automatic retries can resume

Recovery safety caps (why)

First-pass review caught three failure modes in the naive recovery path; this PR addresses each:

1. Ingeminator NPE after reset. resetFailureCount previously wrote meta.lastBuildFailure = null, but Ingeminator.rescheduleFailedBuildsForJob reads it as lastFailure.toMillis() for backoff computation. Once auto-recovery runs every tick, that null deref would crash the scheduler pass before any new editor jobs could be scheduled — i.e. the fix would deadlock the very thing it was meant to unblock. Fixed by (a) writing Timestamp.fromMillis(0) instead of null in resetFailureCount and (b) guarding the Ingeminator with lastBuildFailure ?? Timestamp.fromMillis(0). Defence in depth — the same null path was reachable from the existing admin reset endpoint.
2. Indefinite retry churn. A fundamentally broken build (e.g. an unbuildable editor/baseOs combination) would fail maxFailuresPerBuild (15) times, get auto-reset, fail another 15 times, get reset again, forever — burning Actions minutes and Docker Hub API quota with no escalation. Added meta.recoveryCount (tracked via CiBuilds.incrementRecoveryCount) and Cleaner.maxRecoveryAttempts = 2. After two recovery rounds the build is left at max retries and an alert goes to Discord asking for manual investigation.
3. DockerHub / Firestore hammering. The naive loop processed every maxed-out build for the latest repo every tick, with a DockerHub fetch per build. Aligned with the sibling cleaners by capping at maxBuildsProcessedPerRun (5) per tick; the rest are tagged deferred: in the returned summary and picked up next tick.

Order of operations in scheduleBuildsFromTheQueue

1. CiJobs.markJobsBeforeRepoVersionAsSuperseded — old-repo jobs no longer block the failing-jobs queue
2. Cleaner.recoverMaxedOutFailedBuilds — maxed-out builds either become published (if on DockerHub) or have their failure count reset for the Ingeminator
3. Scheduler runs as before: base image, hub image, ensureThereAreNoFailedJobs (Ingeminator), then buildLatestEditorImages

Recovery runs before the Ingeminator on purpose, so by the time the Ingeminator iterates failed builds, no entry is at max retries.

Firestore index

Added a composite index (status ASC, buildInfo.repoVersion ASC) on ciBuilds for the new getMaxedOutFailedBuildsForRepoVersion / getAllForRepoVersion queries. Without it the first production call would fail with a "create this index" URL.

Known limitations / follow-ups

• Recovery only affects the latest repo version. Older-version failed builds are reachable via the existing admin endpoints; older-version jobs already get marked superseded so they do not block scheduling.
• Recovery only changes build-level state. The parent CiJob stays 'failed' until all its builds reach 'published'. If the failing-job count exceeds maxToleratedFailures (2), ensureThereAreNoFailedJobs still short-circuits before buildLatestEditorImages. This PR fixes the retry-exhausted sub-case; clearing backlogs of healthy-but-failing jobs is still the Ingeminator's responsibility.
• specificTag / friendlyTag written by markBuildAsPublished from recovery are ${baseOs}-${repoVersion} and the major.minor of repoVersion respectively. These do not include the editor version / target platform. This matches the existing healFailedBuildsAlreadyOnDockerHub and cleanUpBuildsThatDidntReportBack cleaners — i.e. an existing inconsistency, worth a follow-up to normalise across all three call sites.
• queueStatus is now world-readable with no auth. Response includes job IDs, statuses and DockerHub digests; write paths (resetFailedBuilds, retryBuild) remain auth-gated. Intentional, called out here for visibility.

Testing

• yarn typecheck

[coderabbitai]

📝 Walkthrough

Walkthrough

Refactoring removes global RepoVersionInfo dependency from Ingeminator and instead retrieves version info per-job; adds maxed-out failed build recovery to Cleaner with DockerHub checks; extends CiBuilds with repo-version-scoped queries; integrates recovery into the scheduler; and updates API endpoints to use new query capabilities.

Changes

Build Queue Refactoring and Recovery

Layer / File(s)	Summary
Repository version-scoped CiBuilds queries functions/src/model/ciBuilds.ts	CiBuilds gains getAllForRepoVersion() and getMaxedOutFailedBuildsForRepoVersion() static methods to query builds filtered by repository version and failure threshold.
Maxed-out failed build recovery service functions/src/logic/buildQueue/cleaner.ts	Cleaner.recoverMaxedOutFailedBuilds(repoVersion) method queries maxed-out failed builds per repo, checks DockerHub image existence, marks published builds with metadata, or resets failure counts, and logs results.
Ingeminator constructor refactoring functions/src/logic/buildQueue/ingeminator.ts	Ingeminator removes instance-level RepoVersionInfo dependency; constructor now accepts only concurrency and GitHub client; rescheduleBuild parses version from per-job jobData.repoVersionInfo.
Build queue scheduling integration functions/src/logic/buildQueue/scheduleBuildsFromTheQueue.ts, functions/src/logic/buildQueue/scheduler.ts	scheduleBuildsFromTheQueue invokes Cleaner.recoverMaxedOutFailedBuilds() before initializing the scheduler; scheduler.ts updates Ingeminator instantiation to remove repoVersionInfo argument.
API endpoint updates functions/src/api/queueStatus.ts, functions/src/api/retryBuild.ts	queueStatus adds CORS headers, OPTIONS handling, optional repoVersion query parameter, and conditional repo-version-scoped build fetching; retryBuild removes RepoVersionInfo import and updates Ingeminator construction.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• game-ci/versioning-backend#92: Refactors retryBuild, Scheduler, and Ingeminator to derive repo version from jobData.repoVersionInfo instead of injected RepoVersionInfo.
• game-ci/versioning-backend#90: Modifies Cleaner to reconcile failed builds by checking DockerHub tags and marking builds as published when images exist.
• game-ci/versioning-backend#85: Adjusts failed-build rescheduling in ingeminator.ts to skip maxed-out builds in the rescheduling path.

Suggested reviewers

• webbertakken
• GabLeRoux

Poem

🐰 A rabbit hops through queues with care,
Per-job versions now found there!
Maxed-out builds, DockerHub's gleam,
Recovery flows—a cleaner dream.
CORS headers, OPTIONS too,
Refactored paths, solid and true! 🎉

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarizes the main changes: auto-recovery of stuck queue builds and CORS/filtering fixes for the queue status API.
Description check	✅ Passed	PR description is comprehensive, well-structured, and covers all required sections with detailed problem statement, changes, and implementation details.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/retry-use-job-repo-version

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}