When enabling "Prevent vulnerable images from running", cannot push an image update

[silviubajenaru]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior:
Jenkins log:

docker push harbor.domain.com/nameserverdns/dev/bind:20260728
The push refers to repository [harbor.domain.com/nameserverdns/dev/bind]
6163cb122752: Waiting
37008765f419: Waiting
827bef93144e: Waiting
c6313663a00a: Waiting
9ac1877b4700: Waiting
ca2678b20700: Waiting
94528cf87bc1: Waiting
fd966f87db4b: Waiting
9ac1877b4700: Waiting
ca2678b20700: Layer already exists
94528cf87bc1: Waiting
6163cb122752: Waiting
37008765f419: Waiting
827bef93144e: Waiting
c6313663a00a: Waiting
37008765f419: Mounted from nameserverdns/docker_update/bind
827bef93144e: Mounted from nameserverdns/docker_update/bind
c6313663a00a: Mounted from nameserverdns/docker_update/bind
9ac1877b4700: Mounted from nameserverdns/docker_update/bind
94528cf87bc1: Mounted from nameserverdns/docker_update/bind
fd966f87db4b: Pushed
6163cb122752: Mounted from nameserverdns/docker_update/bind
20260728: digest: sha256:a081dd45c21dbd13bf4225be4174c9d15894f8116bd7fd71afc7b17b52de36ae size: 856

docker tag harbor.domain.com/nameserverdns/dev/bind:20260728 harbor.domain.com/nameserverdns/dev/bind:88e41d71

docker push harbor.domain.com/nameserverdns/dev/bind:88e41d71
The push refers to repository [harbor.domain.com/nameserverdns/dev/bind]
fd966f87db4b: Waiting
827bef93144e: Waiting
37008765f419: Waiting
9ac1877b4700: Waiting
ca2678b20700: Waiting
c6313663a00a: Waiting
94528cf87bc1: Waiting
6163cb122752: Waiting
827bef93144e: Layer already exists
37008765f419: Layer already exists
9ac1877b4700: Layer already exists
ca2678b20700: Layer already exists
c6313663a00a: Layer already exists
94528cf87bc1: Layer already exists
6163cb122752: Layer already exists
fd966f87db4b: Already exists
unknown: unexpected status from HEAD request to https://harbor.domain.com/v2/nameserverdns/dev/bind/manifests/sha256:90f9ecd1b09d1d122ec06451cb8a858797b710d351062508922414ac33c35664: 412 Precondition Failed

So the first image push works, the image doesn't have any Critical vulnerabilities (the "Prevent vulnerable images from running." is set to Critical, so this should work.

Steps to reproduce the problem:

1. Create a new Project
2. Go to the project's Configuration zone and enable Prevent vulnerable images from running, setting it to Critical
3. Push a non-vulnerable / lower than critical vulnerable image
4. Retag that image as something and push it again to the same repo
5. Error

Versions:
Please specify the versions of following systems.

• harbor version: [2.14.0]
• docker engine version: [29.6.2, build dfc4efb]
• docker-compose version: [v5.3.1]

This seems to be somewhat related to #15056 and maybe #15283 ?

Mentioning the colleague who found this so he's also following this issue: @DanCruceat
Thank you!

[silviubajenaru]

We just updated to Harbor 2.15.2, the problem still persists.

[wy65701436]

Could you confirm whether the Docker client is issuing an HTTP HEAD or GET request to Harbor?

If so, this is expected behavior by design, as policies are evaluated against these requests to enforce security controls.

[silviubajenaru]

Hello,

I have no clue how to assert that. Still, I guess this is not the desired behavior, as it doesn't really make any sense. I am uploading an image that doesn't have critical security issues, and I can't push the same image but with a different tag...?!

[jUDASmILE]

@silviubajenaru do you want to create a new tag for the existing image? If so you can make it via harbor UI. And output of your operation is expected which to push a same image with different tag. you can either push a different image or create a new tag in harbor UI.

[silviubajenaru]

@jUDASmILE something doesn't add up.
I have a pipeline that builds an image. It pushes the image with the commit ID as the tag, and latest as tag. This means that I'm pushing an image like harbor.domain.com/project/imageA:d2461s and then I'm pushing (the same image) as harbor.domain.com/project/imageA:latest. The second push fails, IF the Prevent vulnerable images from running is set. The thing is, if the image would be vulnerable, I'd get to get this warning, because most likely, it needs to pull that image, compare stuff, whatever.
But the image that I'm pushing only has low vulnerabilities, while my prevention option is set to Critical.

If the Prevent vulnerable images from running is not set, I can push my image with both the commit ID and the latest tags, with no errors.

[jUDASmILE]

@silviubajenaru Thanks for the explanation. I tried your scenario in my environment but I couldn't reproduce it. I tried two images which upload the image firstly and then retag it and push the retagged image and succeeded on both.
Hence I created a new project in https://demo.goharbor.io/harbor/projects/7290/repositories, could you please reproduce it in this demo environment? And if it is possible please record a video for the issue reproduction which is easier for us to debug. thanks.

[silviubajenaru]

@jUDASmILE happy to help.

Ok, I was able to reproduce this on your env too:

# docker build -t demo.goharbor.io/23638/test:one . && docker tag demo.goharbor.io/23638/test:one demo.goharbor.io/23638/test:two && docker push demo.goharbor.io/23638/test:one; docker push demo.goharbor.io/23638/test:two
[+] Building 1.1s (6/6) FINISHED                                                                                                                                                                                             docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                   0.0s
 => => transferring dockerfile: 89B                                                                                                                                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:24.04                                                                                                                                                                        0.9s
 => [internal] load .dockerignore                                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                                        0.0s
 => [1/2] FROM docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea                                                                                                                  0.0s
 => => resolve docker.io/library/ubuntu:24.04@sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea                                                                                                                  0.0s
 => CACHED [2/2] RUN apt update && apt upgrade -y                                                                                                                                                                                      0.0s
 => exporting to image                                                                                                                                                                                                                 0.1s
 => => exporting layers                                                                                                                                                                                                                0.0s
 => => exporting manifest sha256:fd1407042eecc3e025c34916c3b3a55b94eb41cf10c996d8ac74f6616141ec8e                                                                                                                                      0.0s
 => => exporting config sha256:3238b5f9e61896e0992c129634d8a49b10e1b9bd3ec9f9b0f25b962d89dc0bfd                                                                                                                                        0.0s
 => => exporting attestation manifest sha256:013d4fb4c343e744543220e891cd3ad35ba252de50651d27ed0ab55315d99c83                                                                                                                          0.0s
 => => exporting manifest list sha256:1f6a9a3ff947636ba79ca3ce486b85269f1ad01b5595cbcfb5dc1cb9e50322b3                                                                                                                                 0.0s
 => => naming to demo.goharbor.io/23638/test:one                                                                                                                                                                                       0.0s
 => => unpacking to demo.goharbor.io/23638/test:one                                                                                                                                                                                    0.0s
The push refers to repository [demo.goharbor.io/23638/test]
966c395d29cb: Layer already exists 
a496e91dfbfc: Pushed 
946c17146b54: Layer already exists 
one: digest: sha256:1f6a9a3ff947636ba79ca3ce486b85269f1ad01b5595cbcfb5dc1cb9e50322b3 size: 855
The push refers to repository [demo.goharbor.io/23638/test]
a496e91dfbfc: Already exists 
946c17146b54: Layer already exists 
966c395d29cb: Layer already exists 
unknown: unexpected status from HEAD request to https://demo.goharbor.io/v2/23638/test/manifests/sha256:fd1407042eecc3e025c34916c3b3a55b94eb41cf10c996d8ac74f6616141ec8e: 412 Precondition Failed

This is all I am doing :D

The dockerfile I used is this:

FROM ubuntu:24.04

RUN apt update && apt upgrade -y

P.S. I have enabled on that repo the following:

• Prevent vulnerable images from running.
  Prevent images with vulnerability severity of Critical and above from being deployed.
• Automatically scan images on push
  Automatically scan images when they are pushed to the project registry.

[jUDASmILE]

@silviubajenaru I tried your steps in a newly created project with same configurations but I can upload the two images successfully...
The project: 23638-2, please check if I setup correctly. I enabled level = Critical and above; Enabled scan on image push.
Could you check your environment and try in another setup see if any difference? thanks.

`<.goharbor.io/23638-2/test:two && docker push demo.goharbor.io/23638-2/test:one; docker push demo.goharbor.io/23638-2/test:two
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
Install the buildx component to build images with BuildKit:
https://docs.docker.com/go/buildx/

Sending build context to Docker daemon 2.048kB
Step 1/2 : FROM ubuntu:24.04
24.04: Pulling from library/ubuntu
966c395d29cb: Pulling fs layer
966c395d29cb: Verifying Checksum
966c395d29cb: Download complete
966c395d29cb: Pull complete
Digest: sha256:561618e2c15bf2397621dd04f96926663a3b5616c189cf7e38db7e82f5c538ea
Status: Downloaded newer image for ubuntu:24.04
---> 045183670ef2
Step 2/2 : RUN apt update && apt upgrade -y
---> Running in 2653150daec2

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
Get:2 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
Get:3 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
Get:4 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [1639 kB]
Get:5 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
Get:6 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1522 kB]
Get:7 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages [50.0 kB]
Get:8 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1143 kB]
Get:9 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
Get:10 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
Get:11 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
Get:12 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
Get:13 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [55.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1469 kB]
Get:15 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [1756 kB]
Get:16 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [2138 kB]
Get:17 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [35.9 kB]
Get:18 http://archive.ubuntu.com/ubuntu noble-backports/multiverse amd64 Packages [671 B]
Get:19 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Packages [48.9 kB]
Fetched 32.1 MB in 3s (9253 kB/s)
Reading package lists...
Building dependency tree...
Reading state information...
1 package can be upgraded. Run 'apt list --upgradable' to see it.

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
The following packages will be upgraded:
libssl3t64
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1942 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 libssl3t64 amd64 3.0.13-0ubuntu3.12 [1942 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 1942 kB in 0s (9136 kB/s)
(Reading database ... 4381 files and directories currently installed.)
Preparing to unpack .../libssl3t64_3.0.13-0ubuntu3.12_amd64.deb ...
Unpacking libssl3t64:amd64 (3.0.13-0ubuntu3.12) over (3.0.13-0ubuntu3.11) ...
Setting up libssl3t64:amd64 (3.0.13-0ubuntu3.12) ...
Processing triggers for libc-bin (2.39-0ubuntu8.8) ...
---> Removed intermediate container 2653150daec2
---> 8521bbee6a22
Successfully built 8521bbee6a22
Successfully tagged demo.goharbor.io/23638-2/test:one
The push refers to repository [demo.goharbor.io/23638-2/test]
897357f89ab3: Pushed
42724e448bf4: Pushed
one: digest: sha256:6c10473420aff106f376e90129241d5fc00f75b781303250bb2046afd7ccf351 size: 741
The push refers to repository [demo.goharbor.io/23638-2/test]
897357f89ab3: Layer already exists
42724e448bf4: Layer already exists
two: digest: sha256:6c10473420aff106f376e90129241d5fc00f75b781303250bb2046afd7ccf351 size: 741`

[silviubajenaru]

Ok so more info:
I was able to reproduce this on some hosts. The difference between the working hosts and the non-working ones is that the ones that aren't able to push the image are using the newer containerd image store: https://docs.docker.com/engine/storage/containerd/
Adding the "debug": true stanza to daemon.json shows that before pushing, a HEAD request is issued to which Harbor replies with 412.

Aug 06 11:33:41 non-working-host dockerd[6513]: time="2026-08-06T11:33:41.768182807+03:00" level=debug msg="do request" digest="sha256:be8794cd00f43f700a3404929995132d1e8d6d0ab6cad2ee2d72b41236f6b865" mediatype=application/vnd.oci.image.manifest.v1+json request.header.accept="application/vnd.oci.image.manifest.v1+json, */*" request.header.user-agent="docker/29.7.1 go/go1.26.5 git-commit/c5b8ce9 kernel/6.8.0-137-generic os/linux arch/amd64 containerd-client/2.3.3+unknown storage-driver/overlayfs UpstreamClient(Docker-Client/29.7.1 \\(linux\\))" request.method=HEAD size=836 url="https://demo.goharbor.io/v2/23638-2/test/manifests/sha256:be8794cd00f43f700a3404929995132d1e8d6d0ab6cad2ee2d72b41236f6b865"
Aug 06 11:33:41 non-working-host dockerd[6513]: time="2026-08-06T11:33:41.845550212+03:00" level=debug msg="fetch response received" digest="sha256:b43e24b94951546a27628b42424bafc5f929ccc019606a389417868d49a1a675" mediatype=application/vnd.oci.image.manifest.v1+json response.header.connection=keep-alive response.header.content-length=328 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 06 Aug 2026 08:33:41 GMT" response.header.set-cookie="sid=9dd762a053dbb9105293ad61f729fb85; Path=/; HttpOnly" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=d0df95d32a7dbc4bcf570bdb7968d1a6 response.status="412 Precondition Failed" size=675 url="https://demo.goharbor.io/v2/23638-2/test/manifests/sha256:b43e24b94951546a27628b42424bafc5f929ccc019606a389417868d49a1a675"
Aug 06 11:33:41 non-working-host dockerd[6513]: time="2026-08-06T11:33:41.845697925+03:00" level=debug msg="unexpected response" digest="sha256:b43e24b94951546a27628b42424bafc5f929ccc019606a389417868d49a1a675" error="unexpected status from HEAD request to https://demo.goharbor.io/v2/23638-2/test/manifests/sha256:b43e24b94951546a27628b42424bafc5f929ccc019606a389417868d49a1a675: 412 Precondition Failed" mediatype=application/vnd.oci.image.manifest.v1+json size=675
Aug 06 11:33:41 non-working-host dockerd[6513]: time="2026-08-06T11:33:41.846150096+03:00" level=debug msg="unable to unmarshal registry error" error="unexpected end of JSON input"

From what I read, that HEAD request is sent to check if the layer is already present...?
Why is Harbor replying with 412? Is this normal?!

[jUDASmILE]

root@lvn-dvm-10-162-42-224 [ ~/demo ]# docker info -f '{{ .DriverStatus }}'
[[driver-type io.containerd.snapshotter.v1]]
root@lvn-dvm-10-162-42-224 [ ~/demo ]# docker --version
Docker version 29.5.2, build 79eb04c
root@lvn-dvm-10-162-42-224 [ ~/demo ]# cat /etc/docker/daemon.json
{
"features": {
"containerd-snapshotter": true
},
"ipv6": true,
"debug": true,
"experimental": true,
"ip6tables": true,
"fixed-cidr-v6": "fd00::/80"
}

Thanks for the updates. I tried to enabled containerd image store per guide you mentioned and retry upload images on two projects 23638-3 and 23638-4. Both succeeded. Anything I missed? Above is the docker configurations of my environment, thanks.

[silviubajenaru]

In theory, that's it.
Something that's weird, all the VMs that have been configured via Puppet's Docker module are having this symptom: https://forge.puppet.com/modules/puppetlabs/docker/readme

[jUDASmILE]

@silviubajenaru how this issue occurred on Puppet's configured vms? the docker push commands are executed in pipeline or manually. I suppose this might a race condition between background scanning (Scan on Push) and containerd's docker push behavior. Once image pushed with first tag the scan will starts, if the scan is still running and another push triggered then harbor think this image is not ready and tread the HEAD request as an unauthorized pull/access attempt on an unscanned/in-progress image and rejects it with 412 Precondition Failed. This is expected behavior of harbor. To resolve this issue you may try to add some sleep between two pushes see if it works.