Skip to content

Availability check blind to stale tag caches: registry returns 200 for tag HEAD but 404 for manifest by digest #626

Description

@MAXxATTAXx

Problem

kuik's active availability check (routing.activeCheck) and monitoring (ClusterImageSetAvailability) both verify image availability by requesting the tag via HEAD /v2/<name>/manifests/<tag>. This is a single-step check.

Container runtimes (containerd, CRI-O, Docker) use a two-step pull path:

  1. GET /v2/<name>/manifests/<tag> — resolve tag to a digest
  2. GET /v2/<name>/manifests/sha256:<digest> — fetch the manifest by digest

Some registries evaluate these two requests independently:

  • JFrog Artifactory with Xray policies — a security policy can block the digest-addressed manifest while the tag lookup still returns 200.
  • Pull-through proxies with stale tag caches (Nexus, Harbor, generic proxies) — the proxy has cached the tag→digest mapping but the upstream manifest is gone; the tag HEAD succeeds against the cache, the digest fetch fails against the upstream.

Symptom

Pods on nodes that do not have the image cached locally enter ImagePullBackOff:

Failed to pull image "registry.example.com/app:v1.2.3":
  content at https://registry.example.com/v2/app/manifests/sha256:abc123… not found

Meanwhile, kuik reports the image as Available and does not attempt to re-mirror or route to an alternative, because the tag-level HEAD check returns 200.

Expected behavior

kuik's availability check should detect the tag/digest inconsistency and report the image as NotFound, triggering the normal fallback/re-mirror path.

Affected setups

  • Artifactory instances with Xray scan policies that block individual manifest digests
  • Pull-through proxy registries (Nexus, Harbor) with stale tag-to-digest cache entries
  • Any registry that enforces access control at the digest level independently of the tag level

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions