Running action with custom image fails randomly with "No such image" error

[stavbernazport]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug, and:

• The documentation does not mention anything about my problem
• There are no open or closed issues that are related to my problem

Description

In order to avoid dockerhub ratelimiting issues, I wanted to change the image being used in setup buildx.

I'm running multi platform builds for my docker images for both amd64 and arm64 architectures.
Each build runs on a Self hosted GitHub Runner with the relevant architecture.

When using the action with a custom image:

- name: Set up Docker Buildx
        id: buildx
        uses: docker/setup-buildx-action@v3
        with:
          platforms: ${{ matrix.platform }}
          driver-opts: |
            image=public.ecr.aws/blinkit/moby/buildkit:v0.13.1

It doesn't always find the image with the following error:

Run docker/setup-buildx-action@v3
Docker info
Buildx version
Inspecting default docker context
Creating a new builder instance
Booting builder
Error: ERROR: Error response from daemon: No such image: public.ecr.aws/blinkit/moby/buildkit:v0.13.1

When fetching the manifest for public.ecr.aws/blinkit/moby/buildkit:v0.13.1 it has both architectures included:

docker manifest inspect public.ecr.aws/blinkit/moby/buildkit:v0.13.1                                                                                (production-eu-cluster-1)
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:42a35accfe38387e58ec1ac4701ae340223c268c3d849b1c8070f0f62b0eca25",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:f91de562e22f9d7e7f15efcbabf61ffcbc44a95b0b87a19b9696d900280448e8",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:1573eb38e4dc6c72a582ccebea1ddc7b0c4b70bb14b5333ed8cbdb85c19942b7",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:798fc4f7c3ead9025b5d14c3f8a2f08a34a693bd0184677f4b4f2f6f6eed7e6d",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:0c8756b6c23c39484ae7ef7bd8f85dc2181a6dbbb5936aafaf5dcbcd1dc76609",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1054,
         "digest": "sha256:3721978d7e0291ef27ff22aaffaf0097060c84365cd76bcf61f3dfa326cf893c",
         "platform": {
            "architecture": "riscv64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:821ca4eeed8561ab47c08bff04b1c5ff657912efc4ce36ad90dbe8b05e71f227",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:64034e0f70cad95772c36fadd4a1a114fbd6463f041ea0dd3b0880107acdea9c",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:c3ef100d291ef8c41f0082ed6468219fd74835677afd4f7072b4357b8c472382",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:d7ee33af4679e9f3c706aa474cbc9945d839f5ea44f939bf3f8952111079d16c",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:88cf103b4a83ba07a92fa444397ad393c116fab8a29364b540bc6edd8839dcef",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 1113,
         "digest": "sha256:d5b7d4ae1d989a056921aa81145e118ada1d1b55331fc1855c64de4988587b41",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

Expected behaviour

When using a custom image it should always find the image if it exists

Actual behaviour

I get the error specific above

Repository URL

No response

Workflow run URL

No response

YAML workflow

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3
  with:
    platforms: ${{ matrix.platform }}
    driver-opts: |
      image=public.ecr.aws/blinkit/moby/buildkit:v0.13.1

Workflow logs

Run docker/setup-buildx-action@v3
Docker info
Buildx version
Inspecting default docker context
Creating a new builder instance
Booting builder
Error: ERROR: Error response from daemon: No such image: public.ecr.aws/blinkit/moby/buildkit:v0.13.1

BuildKit logs

Additional info

No response

[crazy-max]

Can you add these steps right before the - name: Set up Docker Buildx:

- name: Inspect buildkit image
  run: docker buildx imagetools inspect public.ecr.aws/blinkit/moby/buildkit:v0.13.1
- name: Pull buildkit image
  run: docker pull public.ecr.aws/blinkit/moby/buildkit:v0.13.1

And post your Workflow logs please?

[stavbernazport]

Hi @crazy-max, thanks for your response.
I added the error to the workflow logs as well.

Posting all of the log is problematic since it might have some sensitive information. I'll do my best to get more if necessary.

The main issue is that it's inconsistent, sometimes the pull succeeds and sometimes it doesn't.

I'll also try what you asked and update 🙏

[crazy-max]

"Docker info" and other grouped logs seems empty

[santhosh-fit]

@crazy-max We've the same issue, the custom image pull fails intermittently with "No such image" error. Here's our workflow logs. The job is running on Github hosted runner. Any insights?

Click to expand logs

Run docker/setup-buildx-action@v3
  with:
    install: true
    driver-opts: image=public.ecr.aws/***/moby/buildkit:latest          
  
    driver: docker-container
    use: true
    cache-binary: true
    cleanup: true
  env:
    AWS_REGION: ca-central-1
    CI_COMMIT_REF_NAME: ***
    COMMIT_SHA: 6470d[2](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:2)35f48f6911bcf45[3](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:3)00b68793d4405efda0
    DOCKER_CLI_EXPERIMENTAL: enabled
    ECR_REGISTRY: [4](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:4)***.dkr.ecr.ca-central-1.amazonaws.com
    RAILS_ENV: demo
Docker info
  /usr/bin/docker version
  Client: Docker Engine - Community
   Version:           26.1.3
   API version:       1.4[5](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:5)
   Go version:        go1.21.10
   Git commit:        b72abbb
   Built:             Thu May 1[6](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:6) 08:33:35 2024
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:33:35 2024
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          1.[7](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:7).24
    GitCommit:        [8](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:8)8bf19b2105c8b175609[9](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:9)3bee28a01ddc2f97182
   runc:
    Version:          1.2.2
    GitCommit:        v1.2.2-0-g7cb3632
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    26.1.3
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.19.2
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.1
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 0
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 88bf19b2[10](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:10)5c8b17560993bee28a01ddc2f97182
   runc version: v1.2.2-0-g7cb3632
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.8.0-1017-azure
   Operating System: Ubuntu 24.04.1 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 7.753GiB
   Name: fv-az1974-699
   ID: cf286e37-3750-46aa-b63c-6e95f0563703
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    [12](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:12)7.0.0.0/8
   Live Restore Enabled: false
  
Buildx version
  /usr/bin/docker buildx version
  github.com/docker/buildx v0.19.2 1fc5647dc281ca3c2ad5b451aeff2dce84f1dc49
Inspecting default docker context
  [
    {
      "Name": "default",
      "Metadata": {},
      "Endpoints": {
        "docker": {
          "Host": "unix:///var/run/docker.sock",
          "SkipTLSVerify": false
        }
      },
      "TLSMaterial": {},
      "Storage": {
        "MetadataPath": "<IN MEMORY>",
        "TLSPath": "<IN MEMORY>"
      }
    }
  ]
Creating a new builder instance
  /usr/bin/docker buildx create --name builder-5710ee3e-07fb-4679-973c-cd2f4e27d[23](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:24)0 --driver docker-container --driver-opt image=public.ecr.aws/***/moby/buildkit:latest --buildkitd-flags --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host --use
  builder-5710ee3e-07fb-4679-973c-cd2f4e[27](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:28)d230
Booting builder
  /usr/bin/docker buildx inspect --bootstrap --builder builder-5710ee3e-07fb-4679-973c-cd2f4e27d2[30](https://github.com/***/***/actions/runs/***/job/35262050644#step:4:31)
  #1 [internal] booting buildkit
  #1 pulling image public.ecr.aws/***/moby/buildkit:latest
  #1 pulling image public.ecr.aws/***/moby/buildkit:latest 0.5s done
  #1 creating container buildx_buildkit_builder-5710ee3e-07fb-4679-973c-cd2f4e27d2300 done
  #1 ERROR: Error response from daemon: No such image: public.ecr.aws/***/moby/buildkit:latest
  ------
   > [internal] booting buildkit:
  ------
  ERROR: Error response from daemon: No such image: public.ecr.aws/***/moby/buildkit:latest
Error: ERROR: Error response from daemon: No such image: public.ecr.aws/***/moby/buildkit:latest

Image inspect

Run docker buildx imagetools inspect public.ecr.aws/***/moby/buildkit:latest
Name:      public.ecr.aws/***/moby/buildkit:latest
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:86c0ad9d[11](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:12)37c186e9d455912167df20e530bdf7f7c19de802e892bb8ca16552
           
Manifests: 
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:5769c54b98840147b74[12](https://github.com/***/***/actions/runs/12657412376/job/35272119026#step:4:13)8f38fb0b0a049e24b11a75bd81664131edd2854593f
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/amd64
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:2e63a82157adcc534f57eff0b17326d47b96ca74a0e614ac312397c346e505a5
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm/v7
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:cf41e48173ddeb5265761961c94a19d49738d0a2fd3caced40a7a5bf18bf0075
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:e5a615d8148515f20dc7bbdb7b92fcc9daf6a203ba6b3d6fb1b592c96af1f242
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/s390x
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:df12327b4782afaf7e45d880bd997f7896e3096b584965526a341b65f4bfab28
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/ppc64le
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:95a4100ba8fdb5f7df2cc9c48c4c87f7b0f047884a49918c9f4c2b31894daadd
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/riscv64
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:e0b68b7b3198a88ae3947536a53c4d866c5f2b7fef37ef4d6c2ba5a3f8fbb080
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:5769c54b98840147b74128f38fb0b0a049e24b11a75bd81664[13](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:14)1edd2854593f
    vnd.docker.reference.type:   attestation-manifest
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:a98e83116b53c0f5d35a85e5a2a77a719a3eb7da079c084a1c26dfb04c5c4f8e
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:2e63a82157adcc534f57eff0b17326d47b96ca74a0e6[14](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:15)ac312397c346e505a5
    vnd.docker.reference.type:   attestation-manifest
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:11650104f504795825a249b20109186041e3f1d3feb24b1ce524de3e70a1e380
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.type:   attestation-manifest
    vnd.docker.reference.digest: sha256:cf41e48173ddeb5265761961c94a19d49738d0a2fd3caced40a7a5bf18bf0075
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:3ee50eacd1b8b505754cbc7efbb9f61314930b30ecd95a34c7b9f045768d8611
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:e5a6[15](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:16)d8148515f20dc7bbdb7b92fcc9daf6a203ba6b3d6fb1b592c96af1f242
    vnd.docker.reference.type:   attestation-manifest
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha256:9[18](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:19)d8fe9442251fb31c8ef774b70841b95344659115bbbcaf6e998a1d622f095
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:df12327b4782afaf7e45d880bd997f7896e3096b584965526a341b65f4bfab28
    vnd.docker.reference.type:   attestation-manifest
               
  Name:        public.ecr.aws/***/moby/buildkit:latest@sha[25](https://github.com/***/***/actions/runs/***/job/35272119026#step:4:26)6:e79334da4e06ddafe345092dad60cbbc1069cdd7d4073b0302bce45ad667428a
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    unknown/unknown
  Annotations: 
    vnd.docker.reference.digest: sha256:95a4100ba8fdb5f7df2cc9c48c4c87f7b0f047884a49918c9f4c2b31894daadd
    vnd.docker.reference.type:   attestation-manifest

[pawel-kosmala]

I have the same but it has been caused by:
Error: ERROR: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

[lishchinsky]

Mine does not download either correct and incorrect image with this driver-opts input, it always pulls moby/buildkit image

Booting builder
/usr/bin/docker buildx inspect --bootstrap
#1 [internal] booting buildkit
#1 pulling image moby/buildkit:buildx-stable-1

Just does not respect this input

UPD. It happens on self-hosted runner, on GH it is ok. I have no idea why, the job is exactly the same, except "runs-on" value.
UPD2. Somehow on self-hosted with the same workflow config in the log this action shows all my custom options in the "Run" section, but options are absent in the "Creating a new builder instance" log section. On the GH runner log section "Creating a new builder instance" contains all my options.

[crazy-max]

UPD. It happens on self-hosted runner, on GH it is ok. I have no idea why, the job is exactly the same, except "runs-on" value.
UPD2. Somehow on self-hosted with the same workflow config in the log this action shows all my custom options in the "Run" section, but options are absent in the "Creating a new builder instance" log section. On the GH runner log section "Creating a new builder instance" contains all my options.

@lishchinsky This might be because you have an old version of Buildx on your self-hosted runner.

Error: ERROR: Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

@pawel-kosmala Looks like you got rate-limited, you need to login before the Setup buildx step to avoid that. GitHub public runners don't have rate limitation but self-hosted runners out of GitHub infrastructure might be rate limited.

Hi @crazy-max, thanks for your response. I added the error to the workflow logs as well.

Posting all of the log is problematic since it might have some sensitive information. I'll do my best to get more if necessary.

The main issue is that it's inconsistent, sometimes the pull succeeds and sometimes it doesn't.

I'll also try what you asked and update 🙏

@santhosh-fit @stavbernazport This step works?:

- name: Pull buildkit image
  run: docker pull public.ecr.aws/blinkit/moby/buildkit:v0.13.1