Is it possible to use AWS ECR as repository for K3S running on AWS EC2

[urupaud]

We have a k3s cluster which is running on AWS ec2, we want to deploy containers into this cluster using images in our AWS ECR, is this possible ?

[HuJake]

We also want to ask this question.

[brandond]

It appears that containerd does not support external credential helpers like docker does. See: https://github.com/containerd/cri/issues/1131. This would prevent you from being able to authenticate to ECR when using the default containerd backend.

[HuJake]

@brandond thanks your reply my question.

We have changed it to docker but after setting the following flag

k3s server --docker --kubelet-arg = "cloud-provider = external" --kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/ availability-zone) / $ (curl -s http://169.254.169.254/latest/meta-data/instance-id) "

Seems unable to solve..
It seems there is no information to refer to

[bigbohne]

not nice but you can always generate the /etc/rancher/k3s/registries.yaml using crontab ...

[brandond]

@HuJake did you actually install the out-of-tree aws cloud provider, set up the aws credential helper, and configure docker registry auth to use the helper?

@bigbohne will it actually re-read it, or is it just loaded on startup? I couldn't tell.

[HuJake]

@brandond

I simply set the relevant flag,

k3s server 
--docker
--kubelet-arg = "cloud-provider = external" \
--kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)/$(curl -s http: / /169.254.169.254/latest/meta-data/instance-id) "

I'm want to use [aws cloud provider] but not sure how to apply related yaml?
Are there references to successful cases?
thanks

[brandond]

@HuJake All that does is set the ID. If you want a full cloud provider you can find deployment documentation at https://github.com/kubernetes/cloud-provider-aws. For authenticating Docker to ECR for image pulls you would want https://github.com/awslabs/amazon-ecr-credential-helper.

[bigbohne]

With docker as container engine this is easily possible using the ecr-credential-helper. But what about containerd? Or is it a design decision to not include plugable authenticators?

[brandond]

The containerd folks seem to have philosophical issues that preclude them from adding support for pluggable authentication.

[bigbohne]

Than it is what it is ...

[brandond]

@bigbohne see https://github.com/containerd/cri/issues/1131#issuecomment-631633816

[canadiannomad]

will it actually re-read it, or is it just loaded on startup? I couldn't tell.

I'm wondering the same... I can easily write a cronjob to grab new creds, but restarting k3s every time....?

[ghost]

We are running k3s with docker as container engine here. I installed and configured the amazon-ecr-credential-helper.
I can successfully docker pull images from our private ECR repo as a regular user, but k3s itself keeps complaining with ErrImagePull.

A kubectl describe pod shows: Failed to pull image "***": rpc error: code = Unknown desc = Error response from daemon: Get https://***: no basic auth credentials

So my understanding of this is that the kubelet service (running inside a docker container) cannot pull the image because it does not know about the credential helper. Any pointers on how to overcome this?

[brandond]

Probably related: awslabs/amazon-ecr-credential-helper#210

[ghost]

I was able to solve/work around this by deploying registry-creds in my cluster (this is basically what the registry-creds addon in Minikube is doing)

[KennyReeldata]

The "registry creds" project didn't work for me, I ran into upmc-enterprises/registry-creds#97. However, I was able to get auth working with K3s by slightly editing this script for my needs: https://stackoverflow.com/a/55658863
Requires the aws cli to be installed on your servers

[johnlane]

Just in case it's useful.... I'm running k3s locally with the docker backend and I was able to pull from ecr using information from here and here.

[mandrean]

@urupaud I got it working by:

1. $ git clone github.com/upmc-enterprises/registry-cred
2. Create a new RBAC manifest, k8s/rbac.yaml:

kind: ServiceAccount
metadata:
  name: registry-creds
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: registry-creds
subjects:
- kind: ServiceAccount
  name: registry-creds
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

3. Edit k8s/deployment.yaml and secret.yaml to suit your needs. For me, I removed everything ACR/GCR/DPR and only kept the ECR stuff. I also added the reference to the new service account and added a missing env for AWS_SESSION_TOKEN:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-creds
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      name: registry-creds
  template:
    metadata:
      labels:
        name: registry-creds
    spec:
      serviceAccountName: registry-creds
      containers:
      - image: upmcenterprises/registry-creds:1.10
        name: registry-creds
        imagePullPolicy: Always
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_SESSION_TOKEN
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SESSION_TOKEN
          - name: awsaccount
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-account
          - name: awsregion
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-region
          # - name: aws_assume_role
          #   valueFrom:
          #     secretKeyRef:
          #       name: registry-creds-ecr
          #       key: aws-assume-role

3. $ kubectl apply -f k8s/
4. Add awsecr-cred as an image pull secret to any deployment using images in ECR:

...
imagePullSecrets:
  - name: awsecr-cred
...

5. $ kubectl logs -l name=registry-creds -n kube-system --tail 1000 | grep "Updated secret awsecr-cred" should return something like:

time="2021-02-18T12:58:12Z" level=info msg="Updated secret awsecr-cred in namespace default"

[moonape1226]

@mandrean
Your solution above works like a charm on my on-premises K3S cluster. Thanks a lot.

[mmaybeno]

Something to clarify for those using k3s and containerd (instead of docker). These are the steps that allowed me to use https://github.com/upmc-enterprises/registry-creds and AWS container registry.

1. On each node, create the k3s directory and registries.yaml file.
   a. sudo mkdir /etc/rancher/k3s
   b. sudo touch /etc/rancher/k3s/registries.yaml
2. Edit the registries.yaml file with your registry info. Change your registry-name to how you see fit. Replace [ACCOUNT_ID] and [AWS_REGION] with your information.

mirrors:
  registry-name:
    endpoint:
      - "http://[ACCOUNT_ID].dkr.ecr.[AWS_REGION].amazonaws.com"

3. Restart the node agent or main service.
   a. Main Node: sudo systemctl restart k3s
   b. Agent Node: sudo systemctl restart k3s-agent.service
4. Proceed to deploy registry-cred as noted by @mandrean.

[garry-t]

could you provide me guide hot to make it work.
I have followed your doc and @mandrean , pod says Successfully got secret for provider awsecr-cred after trying 1 time(s) But i dont see created secret at any namespace only gcr-secret , what I missed?

[mmaybeno]

It's been awhile since I've used this. But from what I remember you should be able to pull images once k3s gets the secret. Have you tried pulling an image from your private ecr? :

sudo crictl pull {aws_service_id}.dkr.ecr.{region}.amazonaws.com/{image}:{tag}

Obviously if you have a deployment file with the image it should pull as it is deployed. But you can test it initially by manually doing it.

[garry-t]

Thanks for your answer, I found another project. And seems like it is worked at least I see secret in each ns with credentials.
Exact problem was to get k3s secret, because registry-cred doesnt work for me
https://nabsul.github.io/helm this what I used

[mmaybeno]

Glad you found a solution. I'll have to keep this new project in mind if I end up having to do this again :).

[damonmaria]

Before I head off down a dead end, has anyone had experience trying a kubelet image credential provider with k3s? It's alpha but still, probably the future.

[damonmaria]

Back then I couldn't find the add ons needed. So I went with just doing a cron job.

[imavroukakis]

oh interesting, thanks for the response! Would it be too much of an imposition for you to share an example?

[damonmaria]

Ours is actually fairly unique because we use AWS SSM Agent (outside of AWS) and therefore have temporary AWS credentials on the machine. So we mount those credentials and use them. All the cron examples I found store AWS creds in secrets which is more likely how you'll be doing it. I think this is what I based mine on: https://artifacthub.io/packages/helm/architectminds/aws-ecr-credential

[imavroukakis]

Thanks again @damonmaria. I came across https://artifacthub.io/packages/helm/architectminds/aws-ecr-credential and it would have worked perfectly out of the box for us, but we are on arm architecture.

[pinkplus]

I tried this approach. It worked! The only difficulty is that I needed to checkout https://github.com/kubernetes/cloud-provider-aws and compile ecr-credential-provider manually.

[juanbrein]

Here is a quick snippet that will automate ecr login on k3s. Just bear in mind it will overwrite the registries.yaml file if you run it like it is. Make sure you set your AWS credentials variables.

#!/bin/bash

ACCOUNT=$(aws sts get-caller-identity|jq -r .Account)
REGISTRY=${ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com
PASSWORD=$(aws ecr get-login-password)

cat <<EOF |sudo tee /etc/rancher/k3s/registries.yaml
mirrors:
  docker.io:
    endpoint:
      - "https://${REGISTRY}"
configs:
  "${REGISTRY}":
    auth:
      username: AWS
      password: ${PASSWORD}
EOF

systemctl restart k3s

Does a lot of assumptions but you should get the idea.

[ekquasar]

Has anyone tried nerdctl - looks like they have the Docker CLI UX for logging in to ECR via AWS CLI:

$ aws ecr get-login-password --region <REGION> | nerdctl login --username AWS --password-stdin <AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com
Login Succeeded

https://github.com/containerd/nerdctl/blob/main/docs/registry.md#amazon-elastic-container-registry-ecr

[tsturzl]

Setting this as a host cronjob was the quickest way I could get things working.

[gilad-aperio]

I've managed to configure this by having the kubelet use the ECR image credential provider. Here's how to do it:

Note: This is using the dntosas/ecr-credential-provider community fork as the official binaries are only available up to v1.29.0. The fork's latest release at the time of writing is compatible with Kubernetes 1.33.

wget https://github.com/dntosas/ecr-credential-provider/releases/download/v1.2.0/ecr-credential-provider-linux-amd64
chmod +x ecr-credential-provider-linux-amd64
mv ecr-credential-provider-linux-amd64 /usr/local/bin/ecr-credential-provider

Note: Replace the AWS credentials and region with the actual values.

mkdir -p /etc/rancher/k3s
tee /etc/rancher/k3s/ecr-credential-provider-config.yaml > /dev/null <<'EOF'
apiVersion: kubelet.config.k8s.io/v1
kind: CredentialProviderConfig
providers:
  - name: ecr-credential-provider
    matchImages:
      - "*.dkr.ecr.*.amazonaws.com"
    defaultCacheDuration: "12h"
    apiVersion: credentialprovider.kubelet.k8s.io/v1
    env:
      - name: AWS_REGION
        value: <AWS_REGION>
      - name: AWS_ACCESS_KEY_ID
        value: <AWS_ACCESS_KEY_ID>
      - name: AWS_SECRET_ACCESS_KEY
        value: <AWS_SECRET_ACCESS_KEY>
EOF

chmod 600 /etc/rancher/k3s/ecr-credential-provider-config.yaml

Installing k3s with kubelet args. These can be added to already-installed nodes by adding these arguments to the k3s service command.

curl -sfL https://get.k3s.io | sh -s - \
  --kubelet-arg="image-credential-provider-config=/etc/rancher/k3s/ecr-credential-provider-config.yaml" \
  --kubelet-arg="image-credential-provider-bin-dir=/usr/local/bin"

[garry-t]

I used for this case https://nabsul.github.io/helm