[Bug]: docker pull specific tag will trigger sync even if images exists locally

[TeCHiScy]

zot version

v2.1.13

Describe the bug

For example, when I use docker pull zot.mydomain.com/repo1/image1:latest to pull image1 from zot registry, the log at zot shows that it tries to sync image from external sources, even if /repo1/image1:latest locally exists.

│ {"time":"2026-01-21T02:50:05.611280445Z","level":"info","message":"trying to get updated image by syncing on demand","repository":"repo1/image1","referenc │
│ e":"latest","caller":"zotregistry.dev/zot/v2/pkg/api/routes.go:2182","func":"zotregistry.dev/zot/v2/pkg/api.getImageManifest","goroutine":3810}             │
│ {"time":"2026-01-21T02:50:05.611345174Z","level":"debug","message":"starting on-demand image sync","repo":"repo1/image1","reference":"latest","serviceID": │
│ 0,"timeout":10800000000000,"caller":"zotregistry.dev/zot/v2/pkg/extensions/sync/on_demand.go:193","func":"zotregistry.dev/zot/v2/pkg/extensions/sync.(*Base │
│ OnDemand).syncImage","goroutine":3807}                                                                                                                      │
│ {"time":"2026-01-21T02:50:05.611387652Z","level":"info","message":"will not sync image, filtered out by content","remote":"registry-1.docker.io","repo":"de │
│ vops/k8sctl","reference":"latest","caller":"zotregistry.dev/zot/v2/pkg/extensions/sync/service.go:306","func":"zotregistry.dev/zot/v2/pkg/extensions/sync.( │
│ *BaseService).SyncImage","goroutine":3807}                                                                                                                  │
│ {"time":"2026-01-21T02:50:05.611425932Z","level":"error","message":"failed to sync image","error":"image is filtered out by sync config","repository":"devo │
│ ps/k8sctl","reference":"latest","caller":"zotregistry.dev/zot/v2/pkg/api/routes.go:2186","func":"zotregistry.dev/zot/v2/pkg/api.getImageManifest","goroutin │
│ e":3810}

The reason is that getImageManifest preferentially uses godigest to check if reference is of format sha256:xxxxxx. But in this case, it is actually latest. This will cause godigest to fail, and fallback to sync externally, if sync extension was enabled.

zot/pkg/api/routes.go

Lines 2172 to 2180 in 088914b

	_, digestErr := godigest.Parse(reference)
	if digestErr == nil {
	// if it's a digest then return local cached image, if not found and sync enabled, then try to sync
	content, digest, mediaType, err := imgStore.GetImageManifest(name, reference)
	if err == nil || !syncEnabled {
	return content, digest, mediaType, err
	}
	}

Currently, my workaround is to configure allow lists in sync extension via contents, so that to avoid local (also private) images to trigger sync. Then, as the logic goes on, it will fallbacks (again) to find the referenced image locally (which will success), as in:

zot/pkg/api/routes.go

Line 2191 in 088914b

return imgStore.GetImageManifest(name, reference)

In my use case, the workaround is not convience, because I want zot to be a proxy registry for all images that not exists locally (private). This can not be configured via contents since it only supports glob syntax, which is essentially a allowlist.

Moreover, when local images was triggered to sync externally, it will introduce non-negligible latency in docker pull (also kubelet pull) operation, even causes k8s ImagePullBackOff, when network condition to upstream registry was poor.

My thought on this issue is:

• Introduce some mechanism to exclude repos in sync extension, or,
• Find reference locally before triggers sync (i.e., not validate digest via godiest preferentially). If needed, a configuration can also be added for backward compatibility.

In conclude, my question is:

• Is this behavior by design?
• If there's any suggestion to improve the behavior, I can take time to draft a PR.

To reproduce

1. Configuration
2. Client tool used
3. Seen error

Expected behavior

No response

Screenshots

No response

Additional context

No response

[andaaron]

The behavior is by design. Let's consider the following scenario.

1. Someone pushed manifest 1 as myrepo:latest on the upstream
2. You sync manifest 1 as myrepo:latest locally
3. Someone pushed manifest 2 as myrepo:latest on the upstream
   At this point myrepo:latest is out of sync locally vs upstream. So your local latest is not really latest.

So we have this logic which is supposed to do a HEAD request (if I remember correctly) and verify the local digest of the manifest matches the remote digest for myrepo:latest before serving it to the client.

However this check on image pull only happens if onDemand is enabled, see https://github.com/project-zot/zot/blob/main/pkg/api/routes.go#L2181. If onDemand is disable you could theoretically sync the images periodically and serve them later when the clients pull them.

[lbsoong]

I also use v2.1.13 and have the same problem with a specific container and tag: https://hub.docker.com/layers/testcontainers/ryuk/0.13.0
This problem does not apply to all images I proxied but in this case the complete image is downloaded again.
From your answer and the following log entries I assume that the remote digest is not retrieved correctly:

{
    "time": "2026-01-22T09:48:21.049625429Z",
    "level": "info",
    "message": "trying to get updated image by syncing on demand",
    "repository": "docker.io/testcontainers/ryuk",
    "reference": "0.13.0",
    "caller": "zotregistry.dev/zot/v2/pkg/api/routes.go:2182",
    "func": "zotregistry.dev/zot/v2/pkg/api.getImageManifest",
    "goroutine": 3128029
}
{
    "time": "2026-01-22T09:48:21.0497543Z",
    "level": "info",
    "message": "sync: syncing image",
    "remote": "registry-1.docker.io",
    "repo": "docker.io/testcontainers/ryuk",
    "reference": "0.13.0",
    "caller": "zotregistry.dev/zot/v2/pkg/extensions/sync/service.go:313",
    "func": "zotregistry.dev/zot/v2/pkg/extensions/sync.(*BaseService).SyncImage",
    "goroutine": 3128082
}
{
    "time": "2026-01-22T09:48:22.517079417Z",
    "level": "info",
    "message": "remote image digest changed, syncing again",
    "repo": "docker.io/testcontainers/ryuk",
    "reference": "0.13.0",
    "localDigest": "sha256:6bd23f44c17142d808b7ea3c0455d80de3371409776f38434fe2a300481d05c4",
    "remoteDigest": "",
    "caller": "zotregistry.dev/zot/v2/pkg/extensions/sync/destination.go:74",
    "func": "zotregistry.dev/zot/v2/pkg/extensions/sync.(*DestinationRegistry).CanSkipImage",
    "goroutine": 3128082
}
{
    "time": "2026-01-22T09:48:22.517157018Z",
    "level": "info",
    "message": "syncing image",
    "remote image": "docker.io/testcontainers/ryuk:0.13.0",
    "local image": "docker.io/testcontainers/ryuk:0.13.0",
    "caller": "zotregistry.dev/zot/v2/pkg/extensions/sync/service.go:491",
    "func": "zotregistry.dev/zot/v2/pkg/extensions/sync.(*BaseService).syncRef",
    "goroutine": 3128082
}
{
    "time": "2026-01-22T09:48:22.810979845Z",
    "level": "info",
    "message": "Copy layer",
    "source": "docker.io/testcontainers/ryuk@sha256:fa81566cb9a09b2cd72a703e17160c6cf6906a22d4eeae716ce7f0f42053efdb",
    "target": "ocidir:///tmp/sync/docker.io/testcontainers/ryuk/.sync/050f4080-07bc-4d85-ac99-20f3c2ca3e90/docker.io/testcontainers/ryuk@sha256:fa81566cb9a09b2cd72a703e17160c6cf6906a22d4eeae716ce7f0f42053efdb",
    "layer": "sha256:d46ab48eaf906060957934cd71c3294990c8d12093fd206474c27d87a4122bb7",
    "goroutine": 3128166,
    "caller": "github.com/regclient/regclient@v0.11.1/image.go:741",
    "func": "github.com/regclient/regclient.(*RegClient).imageCopyOpt.func5"
}
....

[andaaron]

Do you by any chance pull a docker mediatype and covert locally to OCI? If that is the case the digests don't match, so this results in always pulling the entire image.

That docker mediatypes are converted to OCI by default if you have this setting https://github.com/project-zot/zot/blob/main/examples/config-docker-compat-sync.json#L10.
And if you also set https://github.com/project-zot/zot/blob/main/examples/config-docker-compat-sync.json#L28 the docker meditypes are no longer converted.
See the notes at the beginning of this article https://zotregistry.dev/v2.1.12/articles/mirroring/

[lbsoong]

@andaaron Thank's, that was exactly my issue! It works now.