401 Unauthorized pulling from private registry

[anthony-zawacki]

Issue #16 is closed indicating that public/private registries are supported, but project-copacetic/copa-action@main fails to be able to pull from an ECR registry. I've logged into the registry every way in the github action that I could think to do it. As far as I understand, there are no parameters to the buildx container that runs, so I'm not sure how it's supposed to authenticate with the registry.

The documentation at https://github.com/project-copacetic/copa-action?tab=readme-ov-file#ways-to-connect-to-buildkit seems to indicate buildx (Option 1) is the only supported way to connect to private registries.

Selected github action logs:
Example of being able to pull the image locally, can be done as a regular user or as super use, both are successful:

Run sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
  sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
  shell: /usr/bin/bash -e {0}
  env:
    TAG: [2](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:2)5.0.1
    REGISTRY: ***.dkr.ecr.us-east-1.amazonaws.com
    REPOSITORY: ironbank/opensource/keycloak/keycloak
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
25.0.1-0: Pulling from ironbank/opensource/keycloak/keycloak
e[3](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:3)94ea8406c7: Pulling fs layer
0a[4](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:4)84c292e0f: Pulling fs layer
e2bc109f1f18: Pulling fs layer
f32fd4c317d8: Pulling fs layer
f32fd4c317d8: Waiting
0a484c292e0f: Verifying Checksum
0a484c292e0f: Download complete
e394ea8406c7: Verifying Checksum
e394ea8406c7: Download complete
e394ea8406c7: Pull complete
0a484c292e0f: Pull complete
e2bc109f1f18: Verifying Checksum
e2bc109f1f18: Download complete
f32fd4c317d8: Verifying Checksum
f32fd4c317d8: Download complete
e2bc109f1f18: Pull complete
f32fd4c317d8: Pull complete
Digest: sha2[5](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:5)6:a030889378d9a2c981f245ec35974[6](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:6:6)37b431796b84e0afca4693c7008e3f3df6
Status: Downloaded newer image for ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0

Running the copa action fails with 401 unauthorized:

Run project-copacetic/copa-action@main
  with:
    image: ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:[2](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:2)5.0.1-0
    image-report: report.json
    patched-tag: patched
    timeout: 5m
    output: vex.json
    format: openvex
  env:
    TAG: 25.0.1
    REGISTRY: ***.dkr.ecr.us-east-1.amazonaws.com
    REPOSITORY: ironbank/opensource/keycloak/keycloak
    AWS_DEFAULT_REGION: us-east-1
    AWS_REGION: us-east-1
    AWS_ACCESS_KEY_ID: ***
    AWS_SECRET_ACCESS_KEY: ***
Run # check for copa version input, else use latest
Unable to find image 'ghcr.io/project-copacetic/copa-action:v0.7.0' locally
v0.7.0: Pulling from project-copacetic/copa-action
f11c1adaa26e: Pulling fs layer
c802ad29fa74: Pulling fs layer
fd6e[3](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:3)8fed636: Pulling fs layer
d3d5f8ca516f: Pulling fs layer
d3d5f8ca516f: Waiting
c802ad29fa7[4](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:4): Verifying Checksum
c802ad29fa74: Download complete
f11c1adaa26e: Verifying Checksum
f11c1adaa26e: Download complete
d3d[5](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:5)f8ca516f: Verifying Checksum
d3d5f8ca51[6](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:6)f: Download complete
f11c1adaa26e: Pull complete
c802ad29fa[7](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:7)4: Pull complete
fd6e3[8](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:8)fed636: Verifying Checksum
fd6e38fed636: Download complete
fd6e38fed636: Pull complete
d3d5f8ca516f: Pull complete
Digest: sha256:1242566dfddc06682ce27d14e55d6[9](https://github.com/paradyme-management/copa-keycloak/actions/runs/10099241387/job/27928012790#step:9:9)86846cc3e176d43c72c313e21366bb846a
Status: Downloaded newer image for ghcr.io/project-copacetic/copa-action:v0.7.0
+ image=***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
+ report=report.json
+ patched_tag=patched
+ timeout=5m
+ connection_format=buildx
+ format=openvex
+ output_file=vex.json
+ echo ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
+ cut -d: -f1
+ image_no_tag=***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak
+ [ -z vex.json ]
+ output=--format openvex --output ./data/vex.json
+ docker buildx create --name=copa-action
copa-action
+ docker buildx use --default copa-action
+ connection=--addr buildx://copa-action
+ copa patch -i ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 -r ./data/report.json -t patched --addr buildx://copa-action --timeout 5m --format openvex --output ./data/vex.json
#1 resolve image config for docker-image://***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0
#1 ERROR: unexpected status from HEAD request to https://***.dkr.ecr.us-east-1.amazonaws.com/v2/ironbank/opensource/keycloak/keycloak/manifests/25.0.1-0: 401 Unauthorized
Error: unexpected status from HEAD request to https://***.dkr.ecr.us-east-1.amazonaws.com/v2/ironbank/opensource/keycloak/keycloak/manifests/25.0.1-0: 401 Unauthorized
+ echo Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
+ exit 1
Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
Error: Process completed with exit code 1.

This is a private registry and private git repository, but the full contents of the github workflow is:

name: Docker Image Build

on:
  workflow_dispatch:
    inputs:
      keycloak_tag:
        description: "The keycloak tag to patch.  Assumes that ECR already has the ironbank tag with `-0` appended in to use as a base image."
        required: true


jobs:

  build:

    name: Build Image
    runs-on: ubuntu-latest

    steps:
    - name: Configure environment
      run: |
        echo "TAG=${{ github.event.inputs.keycloak_tag }}" >> $GITHUB_ENV
        echo "REGISTRY=***REDACTED***.dkr.ecr.us-east-1.amazonaws.com" >> $GITHUB_ENV
        echo "REPOSITORY=ironbank/opensource/keycloak/keycloak" >> $GITHUB_ENV
        mkdir -p ${HOME}/.aws
        echo "[default]" >> ${HOME}/.aws/credentials
        AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}
        echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ${HOME}/.aws/credentials
        echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ${HOME}/.aws/credentials
        aws ecr get-login-password --region us-east-1 | sudo docker login --username AWS --password-stdin ***REDACTED***.dkr.ecr.us-east-1.amazonaws.com

    - name: Configure AWS credentials us-east-1
      uses: aws-actions/configure-aws-credentials@v2
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: Login to Amazon ECR us-east-1
      id: us-east-1-ecr
      uses: aws-actions/amazon-ecr-login@v1

    - name: Get the Docker image
      run: |
        sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0

    - name: Generate Trivy Report
      uses: aquasecurity/trivy-action@d43c1f16c00cfd3978dde6c07f4bbcf9eb6993ca # 0.16.1
      with:
        scan-type: "image"
        format: "json"
        output: "report.json"
        ignore-unfixed: true
        vuln-type: "os"
        image-ref: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0

    - name: Check vulnerability count
      id: vuln_count
      run: |
        report_file="report.json"
        vuln_count=$(jq 'if .Results then [.Results[] | select(.Class=="os-pkgs" and .Vulnerabilities!=null) | .Vulnerabilities[]] | length else 0 end' "$report_file")
        echo "vuln_count=$vuln_count" >> $GITHUB_OUTPUT
        echo "vuln_count=$vuln_count"

    - name: Run Copa action
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: copa
      # using main for testing purposes
      # use a tag (such as v1 or v1.0.1) at a bare minimum
      # recommendation is to pin to a digest for security and stability
      # and rely on dependabot for digest/version updates
      uses: project-copacetic/copa-action@main
      with:
        image: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0
        image-report: "report.json"
        patched-tag: "patched"
        timeout: "5m" # optional, default is 5m
        output: vex.json # optional
        format: "openvex" # optional, default is openvex

    - name: Tag Copa image for push
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: tag_copa_img
      run:
          docker tag ${REGISTRY}/${REPOSITORY}:${TAG}-0-patched ${REGISTRY}/${REPOSITORY}:${TAG}

    - name: Tag and push the Docker image to us-east-1
      if: steps.vuln_count.outputs.vuln_count != '0'
      env:
        ECR_REGISTRY: ${{ steps.us-east-1-ecr.outputs.registry }}
      run: |
          docker tag ${REPOSITORY}:${TAG} $ECR_REGISTRY/${REPOSITORY}:${TAG}
          docker push $ECR_REGISTRY/$REPOSITORY:${TAG}

In my case, I have copied the original keycloak image from ironbank to our ECR with a tag of 25.0.1-0 and I'm attempting to patch the image and update the 25.0.1 tag for keycloak with the output from copa. I don't think ironbank images have anything special about them, we could copy registry.access.redhat.com/ubi9/ubi-minimal:9.4-949 to our private ECR as registry.access.redhat.com/ubi9/ubi-minimal:9.4-949-0 and then attempt to run copa against it and have the same results.

[ashnamehrotra]

Hi @anthony-zawacki sorry for the delayed response, are you able to patch this same image locally using copa or does it result in the same error? Also, since you are able to pull the image in the workflow, can you try the containerd image store approach for local images to see if that also results in the same error?

[anthony-zawacki]

This is a touch painful. :)

At the top of my workflow, I've added:

    steps:
    - name: Set up crazy-max's Docker server
      uses: crazy-max/ghaction-setup-docker@v3
      with:
        daemon-config: |
          {
            "debug": true,
            "experimental": true,
            "features": {
              "containerd-snapshotter": true
            }
          }
    - name: Get socket path
      run: |
        url=$(docker context inspect | jq -r .[0].Endpoints.docker.Host)
        socket_path=$(echo "$url" | awk -F// '{print $2}')
        echo "$socket_path"
        echo "SOCKET=$socket_path" >> $GITHUB_ENV
        echo "DOCKER_URL=$url" >> $GITHUB_ENV

However, aquasecurity/trivy-action@0.24.0 runs and builds a local image in docker before these steps run. Trying to use the crazy-max docker server fails on the trivy step because the local image they built as part of the set up to the action doesn't exist in that docker server. To work around this, in my environment configuration section, changed it to log into the ECR registry with both the "standard" docker server as well as the crazy-max docker server:

    - name: Configure environment
      run: |
        echo "TAG=${{ github.event.inputs.keycloak_tag }}" >> $GITHUB_ENV
        echo "REGISTRY=948564350653.dkr.ecr.us-east-1.amazonaws.com" >> $GITHUB_ENV
        echo "REPOSITORY=ironbank/opensource/keycloak/keycloak" >> $GITHUB_ENV
        mkdir -p ${HOME}/.aws
        echo "[default]" >> ${HOME}/.aws/credentials
        AWS_ACCESS_KEY_ID=${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY=${{ secrets.AWS_SECRET_ACCESS_KEY }}
        echo "aws_access_key_id = ${AWS_ACCESS_KEY_ID}" >> ${HOME}/.aws/credentials
        echo "aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}" >> ${HOME}/.aws/credentials
        ECR_PASSWORD="$(aws ecr get-login-password --region us-east-1)"
        echo $ECR_PASSWORD | sudo -s docker -H $DOCKER_URL login --username AWS --password-stdin 948564350653.dkr.ecr.us-east-1.amazonaws.com
        sudo -s docker -H $DOCKER_URL info
        echo $ECR_PASSWORD | sudo docker login --username AWS --password-stdin 948564350653.dkr.ecr.us-east-1.amazonaws.com
        sudo docker info

Using that, I can see both servers, the crazy-max one running 27.2.0 and the "standard" one running 26.1.3.

I updated my image pull section to pull the target image to both docker servers:

    - name: Get the Docker image
      run: |
        sudo -s docker -H $DOCKER_URL pull ${REGISTRY}/${REPOSITORY}:${TAG}-0
        sudo docker pull ${REGISTRY}/${REPOSITORY}:${TAG}-0

This allows trivy to run using the "standard" docker server and copa to run using the crazy-max docker server, both accessing different copies of the same image.

The copa step now looks like:

    - name: Run Copa action
      if: steps.vuln_count.outputs.vuln_count != '0'
      id: copa

      uses: project-copacetic/copa-action@main
      with:
        image: ${{ env.REGISTRY }}/${{ env.REPOSITORY }}:${{ env.TAG }}-0
        image-report: "report.json"
        patched-tag: "patched"
        timeout: "5m" # optional, default is 5m
        output: vex.json # optional
        format: "openvex" # optional, default is openvex
        custom-socket: $SOCKET

This appears to work, I now get to where copa times out trying to update the image:

#12 sh -c /usr/bin/microdnf update python-unversioned-command cups-libs krb5-libs libcurl-minimal orc curl-minimal python3 python3-libs python3-setuptools-wheel && /usr/bin/microdnf clean all
#12 0.243 
#12 0.243 (microdnf:7): librhsm-WARNING **: 15:24:37.597: Found 0 entitlement certificates
#12 0.251 
#12 0.251 (microdnf:7): librhsm-WARNING **: 15:24:37.605: Found 0 entitlement certificates
#12 0.335 Downloading metadata...
#12 0.569 Downloading metadata...
#12 1.081 Downloading metadata...
#12 1.335 Package                                                        Repository               Size
#12 1.335 Upgrading:                                                                                  
#12 1.335  cups-libs-1:2.3.3op2-27.el9_4.x86_64                          ubi-9-baseos-rpms    270.2 kB
#12 1.335   replacing cups-libs-1:2.3.3op2-24.el9.x86_64                                              
#12 1.335  curl-minimal-7.76.1-29.el9_4.1.x86_64                         ubi-9-baseos-rpms    131.8 kB
#12 1.335   replacing curl-minimal-7.76.1-29.el9_4.x86_64                                             
#12 1.335  krb5-libs-1.21.1-2.el9_4.x86_64                               ubi-9-baseos-rpms    788.3 kB
#12 1.335   replacing krb5-libs-1.21.1-1.el9.x86_64                                                   
#12 1.335  libcurl-minimal-7.76.1-29.el9_4.1.x86_64                      ubi-9-baseos-rpms    233.7 kB
#12 1.335   replacing libcurl-minimal-7.76.1-29.el9_4.x86_64                                          
#12 1.335  orc-0.4.31-7.el9_4.x86_64                                     ubi-9-appstream-rpms 191.2 kB
#12 1.335   replacing orc-0.4.31-6.el9.x86_64                                                         
#12 1.335  python-unversioned-command-3.9.18-3.el9_4.5.noarch            ubi-9-appstream-rpms  10.3 kB
#12 1.335   replacing python-unversioned-command-3.9.18-3.el9_4.1.noarch                              
#12 1.335  python3-3.9.18-3.el9_4.5.x86_64                               ubi-9-baseos-rpms     30.3 kB
#12 1.335   replacing python3-3.9.18-3.el9_4.1.x86_64                                                 
#12 1.335  python3-libs-3.9.18-3.el9_4.5.x86_64                          ubi-9-baseos-rpms      8.2 MB
#12 1.335   replacing python3-libs-3.9.18-3.el9_4.1.x86_64                                            
#12 1.335  python3-setuptools-wheel-53.0.0-12.el9_4.1.noarch             ubi-9-baseos-rpms    480.1 kB
#12 1.335    replacing python3-setuptools-wheel-53.0.0-12.el9.noarch                                  
#12 1.335 Transaction Summary:
#12 1.335  Installing:        0 packages
#12 1.335  Reinstalling:      0 packages
#12 1.335  Upgrading:         9 packages
#12 1.335  Obsoleting:        0 packages
#12 1.335  Removing:          0 packages
#12 1.335  Downgrading:       0 packages
#12 1.335 Is this ok [y/N]: time="2024-09-05T15:29:04Z" level=error msg="patch exceeded timeout 5m0s"
Error: patch exceeded timeout 5m0s
+ echo Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
+ exit 1
Error patching image ***.dkr.ecr.us-east-1.amazonaws.com/ironbank/opensource/keycloak/keycloak:25.0.1-0 with copa
Error: Process completed with exit code 1.

This is project-copacetic/copacetic#714 which has been fixed, but not yet released. Is there any facility to run a nightly build of copa of some sort where I could test to see if it actually successfully makes it to the end of the patching, or do we need to wait for 0.7.1 release? I have gotten past the step with a custom-built copa binary where I had fixed the missing -y in the command. I used that in my jenkins pipeline where I ended up opening that issue and moby/buildkit#5244 which is stalled because everyone says the problem is someone else's. However, if the microdnf issue is fixed and copa-action works, I'll not worry about the buildkitd bug and just go this route. :)

[ashnamehrotra]

However, aquasecurity/trivy-action@0.24.0 runs and builds a local image in docker before these steps run. Trying to use the crazy-max docker server fails on the trivy step because the local image they built as part of the set up to the action doesn't exist in that docker server.

This issue seems to be related to #46 and using the --docker-host flag within Trivy. We have also released a new version of copa with microdnf issue resolved :)

Let me know if theres any other issues!

[stevejr]

Hi, I am also seeing the 401 issue. I have tried to use the instructions for running a local docker setup with crazy-max/ghaction-setup-docker@v3 but it appears that action is no-longer available. I assume we can use https://github.com/docker/setup-docker-action instead?

[anthony-zawacki]

Yes, for the build I use to test, I changed:

crazy-max/ghaction-setup-docker@v3
aws-actions/configure-aws-credentials@v2
aws-actions/amazon-ecr-login@v1
aquasecurity/trivy-action@0.24.0

to

docker/setup-docker-action@v4
aws-actions/configure-aws-credentials@v4
aws-actions/amazon-ecr-login@v2
aquasecurity/trivy-action@0.28.0

My build action uses

project-copacetic/copa-action@main

because I was debugging. I didn't change this for my test. No other configuration changes, and the image successfully rebuilt in my environment.

[stevejr]

Thanks @anthony-zawacki. I added a comment to an issue on the setup-docker-action repo as that is not starting on my runner - could be because of using custom self-hosted runners.

So, I tried to remove the registry details from my image so instead of ***.dkr.ecr.us-east-1.amazonaws.com/quay.io/jetstack/cert-manager-ctl:v1.14.4 I pulled that image from ECR, retagged to jetstack/cert-manager-ctl:v1.14.4 but now Copa is trying to pull that from docker.io

Did someone state why Copa isn't first looking if the image is already present locally on the runner that is being used? It seems to always default to try and pull the image from the public/private registry it is stored in.

[anthony-zawacki]

I'm pretty sure it does use the local image if it's already present, I don't see any evidence of copa pulling the image in the logs. Just be aware that there are two docker servers in the environment, and that you need the image in both servers.

The details are in this comment: #47 (comment)

There's the "main" docker server which is used by trivy, and there's the "crazy-max" docker server which is used by copa. Just running docker commands without the -H results in modifications to the "main" docker server.

[stevejr]

This is the error I am seeing after retagging my image so that it is not going to ECR or quay.io:
ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

This is the Copa command:
copa patch -i jetstack/cert-manager-ctl:v1.14.4 -r ./data/report.json -t v1.14.4-1740585106 --addr buildx://copa-action --timeout 5m --format openvex --output ./data/vex.json

It feels like it is doing this because the image - jetstack/cert-manager-ctl:v1.14.4 - is present on the runner but not within the container that Copa is running in - could that be the case?

Is this why we have to use the external docker server so that the image is pulled into its context and that is what Copa is looking at? Strange that Trivy seems to be ok with the docker install on the runner but not Copa.

Am wondering whether to just install Copa directly onto our runner and do everything in a simple shell run command in the workflow.

[anthony-zawacki]

If trivy is running properly, I suspect you're only pulling and retagging the image in the main docker server, and that you need to repeat the actions in the "crazy-max" docker server. If you are pulling the image into both servers and retagging them already, then I am at a loss as to what is going wrong.