Commit 2e129dc
authored
feat: incremental DockerHub reconciliation with cursor and cooldowns (#105)
* feat: reconcile ALL versions via bulk DockerHub tag listing
Replace per-image tag existence check with paginated tags-list API call per repo. Compute expected image set from all EditorVersionInfo records and current RepoVersionInfo, then dispatch retries for any tag missing on DockerHub.
Why: previous reconciler limited to 5 most recent versions (RECENT_VERSIONS_TO_CHECK=5), missing scenarios like 6000.3.17f1 being released after newer 6000.4.x versions were ingested. Older versions with missing images on the current repo version were invisible to reconciliation.
Approach: one paginated GET per unityci repo (base, hub, editor) returns existing tags. Cap retries at MAX_RETRIES_PER_CYCLE=30 to avoid GitHub dispatch overload; remaining missing images retry next cycle.
Tests cover: empty input, all-present case, mixed old/new version coverage, API failure, pagination, retry capping.
* feat: incremental reconciliation with Firestore cursor and cooldowns
Per-cycle work is now bounded and survives across runs:
- VERSIONS_PER_CYCLE=5: process 5 versions per cycle, advance cursor in Firestore, wrap to start after full sweep
- MAX_DISPATCHES_PER_CYCLE=10: hard cap on retry_dispatch calls to GitHub per cycle
- DISPATCH_COOLDOWN_MS=2h: per-tag cooldown prevents re-dispatching the same retry repeatedly
- BASE_HUB_CHECK_INTERVAL_MS=6h: base/hub tags rarely change, so we only re-check every 6h
- MAX_TAG_PAGES_PER_QUERY=3: hard cap on pagination so we never walk an unbounded tag list
- DockerHub tag fetches use server-side name= filter scoped per-version per-OS (small page count)
- Optional DOCKERHUB_RECONCILE_TOKEN env var for authenticated DockerHub requests (5000/6h vs 200/6h anonymous)
- 429 handling: bail without dispatching when rate-limited
- COOLDOWN_RETENTION_MS=7d: prune old cooldown entries from Firestore doc to bound its size
Adds ReconciliationState model backed by reconciliationState/dockerHub Firestore doc.
* test: fix mock state isolation and cursor-wrap assertion
Three test fixes:
- ReconciliationState.load mock now deep-clones state (was returning reference, causing cycle 2 mutations to leak into stateAfterCycle1 snapshot)
- 'all expected tags present' now uses pattern-matching mock so all 4 fetches return matching tags (was mocking 6 sequential responses for 4 calls, causing OS-mismatched first response)
- 'cursor wrap' uses 8 versions instead of 3 so cycle after wrap lands at v4 != v7 (with only 3 versions, wrap processes all and cursor returns to same final version)1 parent 03767d7 commit 2e129dc
3 files changed
Lines changed: 542 additions & 102 deletions
File tree
- functions
- src
- logic/buildQueue
- model
- test
0 commit comments