Tag list pagination not followed for ghcr.io — newer tags past first page invisible to the controller

[dvystrcil]

Summary

argocd-image-updater only sees the first N tags returned by ghcr.io's tags/list endpoint without following pagination, so images whose newer tags fall past the first page stay "stuck" — images_skipped=1 every cycle, images_updated=0.

In our repro the controller saw 100 tags (after 1.x constraint pre-filter, 17 candidates), of which 0 matched the -stable regex — but the registry actually has 223 total tags including 44 in the stable channel, accessible via ?n=N.

Environment

• argocd-image-updater v1.1.1 (deployed via Helm)
• master branch (today's v1.2.1 release) still pins github.com/distribution/distribution/v3 v3.0.0-20230722181636-7b502560cad4 — a 2023-07-22 pre-release commit; v3.0.0 / v3.1.0 / v3.1.1 have shipped since then
• Kubernetes 1.30, Argo CD 3.4.x
• Registry: ghcr.io

Reproduction

ImageUpdater CR:

apiVersion: argocd-image-updater.argoproj.io/v1alpha1
kind: ImageUpdater
metadata:
  name: filebrowser-iu
spec:
  namespace: argocd
  applicationRefs:
    - namePattern: "filebrowser"
      useAnnotations: false
      images:
        - alias: "filebrowser"
          imageName: "ghcr.io/gtsteffaniak/filebrowser:1.x"
          commonUpdateSettings:
            allowTags: 'regexp:^1\.\d+\.\d+-stable$'
            updateStrategy: semver

Currently-deployed image: ghcr.io/gtsteffaniak/filebrowser:1.3.2-stable. Newer tag exists in the registry: 1.3.3-stable.

Debug logs (IMAGE_UPDATER_LOGLEVEL=debug):

level=debug msg="Considering this image for update" image_alias=filebrowser image_name=ghcr.io/gtsteffaniak/filebrowser image_registry=ghcr.io image_tag=1.3.2-stable
level=debug msg="Using version constraint '1.x' when looking for a new tag"
level=debug msg="found 0 from 17 tags eligible for consideration"
level=debug msg="No suitable image tag for upgrade found in list of available tags."
level=info  msg="Processing results: applications=1 images_considered=1 images_skipped=1 images_updated=0 errors=0"

Direct check against ghcr (anonymous bearer token):

# Default request: 100 tags, all 0.x-beta/dev and 1.0.x-beta
$ curl -H "Authorization: Bearer $TOK" https://ghcr.io/v2/gtsteffaniak/filebrowser/tags/list \
    | jq '.tags | length'
100

# With explicit n: 223 total
$ curl -H "Authorization: Bearer $TOK" "https://ghcr.io/v2/gtsteffaniak/filebrowser/tags/list?n=2000" \
    | jq '.tags | length'
223

# Stable-channel tags exist past position 100
$ curl -H "Authorization: Bearer $TOK" "https://ghcr.io/v2/gtsteffaniak/filebrowser/tags/list?n=2000" \
    | jq '.tags[] | select(test("^1\\.\\d+\\.\\d+-stable$"))'
"1.3.3-stable"
"1.3.2-stable"
"1.3.1-stable"
"1.3.0-stable"
"1.2.4-stable"
... (and 1.0.x through 1.2.x-stable variants)

The 17 candidates the controller considered = 100 raw tags pre-filtered to those matching 1.x. None of those 17 are in the stable channel because *-stable only starts appearing later in ghcr's lexicographic tag order — past the 100-tag window.

Root cause hypothesis

registry-scanner/pkg/registry/client.go:Tags() calls tagService.All(ctx) from github.com/distribution/distribution/v3. The implementation in the pinned pre-release commit doesn't follow Link: headers for paginated tag listings against ghcr.io.

Expected vs. actual

• Expected: all 223 tags considered (or a documented tagListLimit knob).
• Actual: silent 100-tag cap, with no surface signal that we're hitting it. Debug log says "found 0 from 17" rather than "fetched 1 of N pages".

Suggested fixes (any one)

1. Bump github.com/distribution/distribution/v3 to v3.1.1 (or later) in both registry-scanner/go.mod and the main go.mod. Verify the new client follows Link: headers for ghcr.io.
2. Add a tag_list_limit (or tag_list_pages) registry option to registries.conf exposing the underlying request to tags/list?n=. Pair with a log line so operators see when the cap is hit.
3. Add a debug log when the unpaginated response size equals a suspect round number (100, 1000, etc.), naming pagination as the likely cause.

Workaround

Manual images: override in the kustomize overlay. Loses the auto-update value but unblocks deploys. Tracked locally for several months on at least one image.

[dkarpele]

github.com/distribution/distribution/v3 upgrade to v3.1.1 is currently blocked by #968 (comment), #968 (comment) and #960 (comment)

[dkarpele]

github.com/distribution/distribution/v3 bumped to v3.1.1 in #1675. If you think this resolves an issue feel free to test against this branch https://github.com/dkarpele/argocd-image-updater/tree/dk-GITOPS-9880.
Thanks