ExcludeDockerImages masks end up in missing layers

[olwe0002]

this is the used config:

- name: Remove all docker images from docker-xyz-local older than 14 days
  rules:
    - rule: Repo
      name: docker-xyz-local
    - rule: DeleteDockerImagesOlderThan
      days: 14
    - rule: ExcludeDockerImages
      masks:
        - image/path/with:tag

I expected the image image/path/with:tag to be excluded from deletion, with all its layers. Is that an wrong assumption with this config?

list.manifest.json of this image (note the digest 877fe4031a674aebb3d006692366fb73badfc36e0d1756964b9d1a7db5b8b553):

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "manifests": [
        {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:a7660fa61e8845d998169dc6c6a2cf801284af6a055ea52b94aad540c68d8926",
        "size": 2199,
        "platform": {
            "architecture": "amd64",
            "os": "linux"
        }
        },
        {
        "mediaType": "application/vnd.oci.image.manifest.v1+json",
        "digest": "sha256:877fe4031a674aebb3d006692366fb73badfc36e0d1756964b9d1a7db5b8b553",
        "size": 566,
        "annotations": {
            "vnd.docker.reference.digest": "sha256:a7660fa61e8845d998169dc6c6a2cf801284af6a055ea52b94aad540c68d8926",
            "vnd.docker.reference.type": "attestation-manifest"
        },
        "platform": {
            "architecture": "unknown",
            "os": "unknown"
        }
        }
    ]
}

the output of the deletion shows the nmatch entry for the image, but also the deletion of its layer:

Add AQL Text - rule: Repo - Apply the policy to one repository.
Add AQL Text - rule: DeleteDockerImagesOlderThan - Removes Docker image older than ``days`` days
Add AQL Text - rule: ExcludeDockerImages - Exclude Docker images by name and tags.
********************************************************************************
Result AQL Query:

items.find(
...
{"path": {"$nmatch": "image/path/with/tag"}}


Found 1093 artifacts
Filter artifacts - rule: Repo - Apply the policy to one repository.
Filter artifacts - rule: DeleteDockerImagesOlderThan - Removes Docker image older than ``days`` days
Filter artifacts - rule: ExcludeDockerImages - Exclude Docker images by name and tags.
Found 1093 artifacts AFTER filtering

DESTROY MODE - delete 'docker-xyz-local/image/path/sha256%3A877fe4031a674aebb3d006692366fb73badfc36e0d1756964b9d1a7db5b8b553 - 1K'
...

After that, the image can't be pulled any more, as the layer is missing in Artifactory. This doesn't happen all the time, only for some images or single layers. How can I avoid this?

[olwe0002]

any update on this?

[allburov]

Likely you need to track that down in your environment, especially if it happens time to time.
Feel free to contribute the fix, we're happy to review that!

[felipecrs]

I have the exact same issue, and I could not get around it yet.

[keplerxd]

Most likely, the issue lies in the format used for storing multi-platform images, such as when using docker buildx bake. The difference is shown in the screenshot.

[christianwaldmann]

I believe ExcludeDockerImages doesn't work correctly for multi-platform images if you specify an exact tag to exclude. That's because Artifactory uses the file list.manifest.json ("fat manifest") for storing such docker images which references the platform-specific Docker images (manifest.json files). This tool searches solely for manifest.json files without considering this case and can therefore delete the platform-specific Docker images which in turn breaks the "fat manifest".

Workaround (if you are okay with excluding all tags)

To avoid this issue, you can exclude all tags instead of a specific one:

     - rule: ExcludeDockerImages
       masks:
-       - image/path/with:tag
+       - image/path/with:*

Background

I've tried to recreate this issue with a very simple multi-platform Docker image.

Steps

1. Dockerfile

FROM alpine
RUN echo "Hello" >> /hello

2. Build and push Docker image

docker buildx build --platform linux/amd64,linux/arm64 --tag my-artifactory-domain/docker/hello:latest --output=type=image,push=true --push .

3. There will be 5 folders in Artifactory

• docker/hello/latest/: list.manifest.json
• docker/hello/sha256:99b3a4337f71dab3cd8912ceb58b044f47d0b4a68dacc415c0430a0bd954ec14/: manifest.json (arm64)
• docker/hello/sha256:d5452882611f6dbb91e3ad0cdd08963a91dbe894bf91fcea25713b773d259ceb/: manifest.json (amd64)
• docker/hello/sha256:aad45c7c910975738af7e4a1a5c1771ce1cfe882e22521307e907c7cc8981e19/: manifest.json (attestation-manifest for amd64)
• docker/hello/sha256:0324a549a899219bd8a94625524e7af9f53f35e8d204eab4f6d284739a786887/: manifest.json (attestation-manifest for arm64)

4. Run artifactory-cleanup with a specific tag excluded:

• config:

- rule: ExcludeDockerImages
  masks:
    - hello:latest

• output:

 Found 4 artifacts AFTER filtering
 DEBUG - we would delete 'docker/hello/sha256%3A99b3a4337f71dab3cd8912ceb58b044f47d0b4a68dacc415c0430a0bd954ec14' (60892fd4b11f0156c3fcd3947571fe6d30436166) - 3M
 DEBUG - we would delete 'docker/hello/sha256%3Ad5452882611f6dbb91e3ad0cdd08963a91dbe894bf91fcea25713b773d259ceb' (fd949137d902de2a5b2f374f4974480ae1d1b2db) - 3M
 DEBUG - we would delete 'docker/hello/sha256%3Aaad45c7c910975738af7e4a1a5c1771ce1cfe882e22521307e907c7cc8981e19' (298d419b57fa7869677d1ab3d9cf04065e08166f) - 1K
 DEBUG - we would delete 'docker/hello/sha256%3A0324a549a899219bd8a94625524e7af9f53f35e8d204eab4f6d284739a786887' (034e8d69548bf0234c69ec52840c2bec0a07f91d) - 1K

• conclusion: This incorrectly deletes the Docker images despite them being listed as excluded.

5. Run artifactory-cleanup with all tags excluded:

• config:

- rule: ExcludeDockerImages
  masks:
    - hello:*

• output:

 Found 0 artifacts AFTER filtering

• conclusion: This does not delete the Docker images and therefore works as intended.

[felipecrs]

Thanks for building a reproduceable example, @christianwaldmann.

I'd just like to mention your workaround is not a real workaround in case users indeed need to filter by some docker tags, which what I need and seems to be what the issue author needs too.

[christianwaldmann]

You're right, this workaround this won't work for everyone. But I still think it's helpful for others to know how to avoid this issue if they are fine with excluding all tags.

The preferred solution would be a proper fix in artifactory-cleanup of course.