Skip to content

Commit 2e129dc

Browse files
authored
feat: incremental DockerHub reconciliation with cursor and cooldowns (#105)
* feat: reconcile ALL versions via bulk DockerHub tag listing Replace per-image tag existence check with paginated tags-list API call per repo. Compute expected image set from all EditorVersionInfo records and current RepoVersionInfo, then dispatch retries for any tag missing on DockerHub. Why: previous reconciler limited to 5 most recent versions (RECENT_VERSIONS_TO_CHECK=5), missing scenarios like 6000.3.17f1 being released after newer 6000.4.x versions were ingested. Older versions with missing images on the current repo version were invisible to reconciliation. Approach: one paginated GET per unityci repo (base, hub, editor) returns existing tags. Cap retries at MAX_RETRIES_PER_CYCLE=30 to avoid GitHub dispatch overload; remaining missing images retry next cycle. Tests cover: empty input, all-present case, mixed old/new version coverage, API failure, pagination, retry capping. * feat: incremental reconciliation with Firestore cursor and cooldowns Per-cycle work is now bounded and survives across runs: - VERSIONS_PER_CYCLE=5: process 5 versions per cycle, advance cursor in Firestore, wrap to start after full sweep - MAX_DISPATCHES_PER_CYCLE=10: hard cap on retry_dispatch calls to GitHub per cycle - DISPATCH_COOLDOWN_MS=2h: per-tag cooldown prevents re-dispatching the same retry repeatedly - BASE_HUB_CHECK_INTERVAL_MS=6h: base/hub tags rarely change, so we only re-check every 6h - MAX_TAG_PAGES_PER_QUERY=3: hard cap on pagination so we never walk an unbounded tag list - DockerHub tag fetches use server-side name= filter scoped per-version per-OS (small page count) - Optional DOCKERHUB_RECONCILE_TOKEN env var for authenticated DockerHub requests (5000/6h vs 200/6h anonymous) - 429 handling: bail without dispatching when rate-limited - COOLDOWN_RETENTION_MS=7d: prune old cooldown entries from Firestore doc to bound its size Adds ReconciliationState model backed by reconciliationState/dockerHub Firestore doc. * test: fix mock state isolation and cursor-wrap assertion Three test fixes: - ReconciliationState.load mock now deep-clones state (was returning reference, causing cycle 2 mutations to leak into stateAfterCycle1 snapshot) - 'all expected tags present' now uses pattern-matching mock so all 4 fetches return matching tags (was mocking 6 sequential responses for 4 calls, causing OS-mismatched first response) - 'cursor wrap' uses 8 versions instead of 3 so cycle after wrap lands at v4 != v7 (with only 3 versions, wrap processes all and cursor returns to same final version)
1 parent 03767d7 commit 2e129dc

3 files changed

Lines changed: 542 additions & 102 deletions

File tree

0 commit comments

Comments
 (0)