Is it possible to use AWS ECR as repository for K3S running on AWS EC2

We have a k3s cluster which is running on AWS ec2, we want to deploy containers into this cluster using images in our AWS ECR, is this possible ?

We also want to ask this question.

It appears that containerd does not support external credential helpers like docker does. See: https://github.com/containerd/cri/issues/1131. This would prevent you from being able to authenticate to ECR when using the default containerd backend.

@brandond

@brandond thanks your reply my question.

We have changed it to docker but after setting the following flag

k3s server --docker --kubelet-arg = "cloud-provider = external" --kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/ availability-zone) / $ (curl -s http://169.254.169.254/latest/meta-data/instance-id) "

Seems unable to solve..
It seems there is no information to refer to

not nice but you can always generate the /etc/rancher/k3s/registries.yaml using crontab ...

@HuJake

@HuJake did you actually install the out-of-tree aws cloud provider, set up the aws credential helper, and configure docker registry auth to use the helper?

@bigbohne will it actually re-read it, or is it just loaded on startup? I couldn't tell.

@brandond

@brandond

I simply set the relevant flag,

k3s server 
--docker
--kubelet-arg = "cloud-provider = external" \
--kubelet-arg = "provider-id = aws: /// $ (curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)/$(curl -s http: / /169.254.169.254/latest/meta-data/instance-id) "

I'm want to use [aws cloud provider] but not sure how to apply related yaml?
Are there references to successful cases?
thanks

@HuJake

@HuJake All that does is set the ID. If you want a full cloud provider you can find deployment documentation at https://github.com/kubernetes/cloud-provider-aws. For authenticating Docker to ECR for image pulls you would want https://github.com/awslabs/amazon-ecr-credential-helper.

With docker as container engine this is easily possible using the ecr-credential-helper. But what about containerd? Or is it a design decision to not include plugable authenticators?

The containerd folks seem to have philosophical issues that preclude them from adding support for pluggable authentication.

Than it is what it is ...

@bigbohne

@bigbohne see https://github.com/containerd/cri/issues/1131#issuecomment-631633816

will it actually re-read it, or is it just loaded on startup? I couldn't tell.

I'm wondering the same... I can easily write a cronjob to grab new creds, but restarting k3s every time....?

Probably related: awslabs/amazon-ecr-credential-helper#210

The "registry creds" project didn't work for me, I ran into upmc-enterprises/registry-creds#97. However, I was able to get auth working with K3s by slightly editing this script for my needs: https://stackoverflow.com/a/55658863
Requires the aws cli to be installed on your servers

Just in case it's useful.... I'm running k3s locally with the docker backend and I was able to pull from ecr using information from here and here.

@urupaud

@urupaud I got it working by:

$ git clone github.com/upmc-enterprises/registry-cred
Create a new RBAC manifest, k8s/rbac.yaml:

kind: ServiceAccount
metadata:
  name: registry-creds
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: registry-creds
subjects:
- kind: ServiceAccount
  name: registry-creds
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: edit
  apiGroup: rbac.authorization.k8s.io

Edit k8s/deployment.yaml and secret.yaml to suit your needs. For me, I removed everything ACR/GCR/DPR and only kept the ECR stuff. I also added the reference to the new service account and added a missing env for AWS_SESSION_TOKEN:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry-creds
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      name: registry-creds
  template:
    metadata:
      labels:
        name: registry-creds
    spec:
      serviceAccountName: registry-creds
      containers:
      - image: upmcenterprises/registry-creds:1.10
        name: registry-creds
        imagePullPolicy: Always
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SECRET_ACCESS_KEY
          - name: AWS_SESSION_TOKEN
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: AWS_SESSION_TOKEN
          - name: awsaccount
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-account
          - name: awsregion
            valueFrom:
              secretKeyRef:
                name: registry-creds-ecr
                key: aws-region
          # - name: aws_assume_role
          #   valueFrom:
          #     secretKeyRef:
          #       name: registry-creds-ecr
          #       key: aws-assume-role

$ kubectl apply -f k8s/
Add awsecr-cred as an image pull secret to any deployment using images in ECR:

...
imagePullSecrets:
  - name: awsecr-cred
...

$ kubectl logs -l name=registry-creds -n kube-system --tail 1000 | grep "Updated secret awsecr-cred" should return something like:

time="2021-02-18T12:58:12Z" level=info msg="Updated secret awsecr-cred in namespace default"

@mandrean

@mandrean
Your solution above works like a charm on my on-premises K3S cluster. Thanks a lot.

Navigation Menu

Repository navigation

Is it possible to use AWS ECR as repository for K3S running on AWS EC2 #1427

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Is it possible to use AWS ECR as repository for K3S running on AWS EC2 #1427

Description

Activity

HuJake commented on May 20, 2020

brandond commented on May 20, 2020

HuJake commented on May 20, 2020

bigbohne commented on May 20, 2020

brandond commented on May 21, 2020

HuJake commented on May 21, 2020

brandond commented on May 22, 2020

bigbohne commented on Jun 23, 2020

brandond commented on Jun 23, 2020

bigbohne commented on Jun 24, 2020

brandond commented on Jun 24, 2020

canadiannomad commented on Jul 30, 2020

brandond commented on Oct 21, 2020

KennyReeldata commented on Nov 26, 2020

johnlane commented on Feb 10, 2021

mandrean commented on Feb 18, 2021

moonape1226 commented on Mar 3, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions