bug(al2023): CredentialProviderConfig doesn't include dualstack endpoints

[paulcdejean]

What happened:

CredentialProviderConfig set in al2023 doesn't provide dualstack endpoints, which makes it unable to pull images if there isn't ipv4 access.

What you expected to happen:

• "ecr-public.aws.com" # dualstack public
• "*.dkr-ecr.*.on.aws" # dualstack private

Are included in the list as they're valid ECR endpoints.

How to reproduce it (as minimally and precisely as possible):

View the CredentialProviderConfig, see that ECR dualstack endpoints are not included.

Environment:

• AWS Region: us-east-2
• Instance Type(s): t4g.xlarge
• Cluster Kubernetes version: 1.34
• Node Kubernetes version: 1.34
• AMI Version: latest

[paulcdejean]

I'd also recommend adding:

  env:
    - name: AWS_USE_DUALSTACK_ENDPOINT
      value: "true"

[mselim00]

@paulcdejean thanks for opening this, I'm looking into how we can support authenticated pulls for the public ECR dualstack endpoint

can you clarify the error you're seeing though? the match rule should already support the private dualstack endpoint, and public ECR does not require authenticated pulls, though unauthenticated pulls are subject to more throttling.

here are all the currently supported endpoints:

amazon-eks-ami/nodeadm/internal/kubelet/image-credential-provider.go

Lines 61 to 72 in d4df9a2

	"*.dkr.ecr.*.amazonaws.com",
	"*.dkr-ecr.*.on.aws",
	"*.dkr.ecr.*.amazonaws.com.cn",
	"*.dkr-ecr.*.on.amazonwebservices.com.cn",
	"*.dkr.ecr-fips.*.amazonaws.com",
	"*.dkr-ecr-fips.*.on.aws",
	"*.dkr.ecr.*.c2s.ic.gov",
	"*.dkr.ecr.*.sc2s.sgov.gov",
	"*.dkr.ecr.*.cloud.adc-e.uk",
	"*.dkr.ecr.*.csp.hci.ic.gov",
	"*.dkr.ecr.*.amazonaws.eu",
	"public.ecr.aws",

[paulcdejean]

public ECR does not require authenticated pulls

Unauthenticated pulls via the dualstack endpoint were not working for me. However with the following config I'm able to successfully pull.

---
kind: CredentialProviderConfig
apiVersion: kubelet.config.k8s.io/v1
providers:
- name: docker-credential-ecr-login
  env:
    - name: AWS_USE_DUALSTACK_ENDPOINT
      value: "true"
  matchImages:
  - "*.dkr.ecr.*.amazonaws.com"
  - "*.dkr-ecr.*.on.aws"
  - "*.dkr.ecr.*.amazonaws.com.cn"
  - "*.dkr-ecr.*.on.amazonwebservices.com.cn"
  - "*.dkr.ecr-fips.*.amazonaws.com"
  - "*.dkr-ecr-fips.*.on.aws"
  - "*.dkr.ecr.*.c2s.ic.gov"
  - "*.dkr.ecr.*.sc2s.sgov.gov"
  - "*.dkr.ecr.*.cloud.adc-e.uk"
  - "*.dkr.ecr.*.csp.hci.ic.gov"
  - "*.dkr.ecr.*.amazonaws.eu"
  - "public.ecr.aws"
  - "ecr-public.aws.com" # dualstack public
  - "*.dkr-ecr.*.on.aws" # dualstack private

  defaultCacheDuration: 12h0m0s
  apiVersion: credentialprovider.kubelet.k8s.io/v1

I'm using a custom AMI to enable this.

[paulcdejean]

  Normal   Scheduled  2m6s                 default-scheduler  Successfully assigned kube-system/aws-node-h7xwq to i-0a1119a5c89eef468.us-east-2.compute.internal
  Normal   Pulling    2m6s                 kubelet            spec.initContainers{aws-vpc-cni-init}: Pulling image "ecr-public.aws.com/eks/amazon-k8s-cni-init:v1.21.1"
  Normal   Pulled     2m3s                 kubelet            spec.initContainers{aws-vpc-cni-init}: Successfully pulled image "ecr-public.aws.com/eks/amazon-k8s-cni-init:v1.21.1" in 2.504s (2.504s including waiting). Image size: 66115609 bytes.

[paulcdejean]

Maybe I lied. I'm seeing this in the logs:

error execing credential provider plugin docker-credential-ecr-login for image ecr-public.aws.com/eks/aws-network-policy-agent: exit status 1

But the image was able to pull. So maybe it was unauthed.

[paulcdejean]

The ecr credential plugin doesn't seem to work with the dualstack endpoint sadly.

[fedora@i-0a1119a5c89eef468 ~]$ echo "public.ecr.aws/eks/aws-network-policy-agent:v1.3.1-eksbuild.1" | docker-credential-ecr-login get
{"ServerURL":"public.ecr.aws/eks/aws-network-policy-agent:v1.3.1-eksbuild.1","Username":"AWS","Secret":"yadayada"}
[fedora@i-0a1119a5c89eef468 ~]$ echo "ecr-public.aws.com/eks/aws-network-policy-agent:v1.3.1-eksbuild.1" | docker-credential-ecr-login get
credentials not found in native keychain
[fedora@i-0a1119a5c89eef468 ~]$ 

[paulcdejean]

https://github.com/awslabs/amazon-ecr-credential-helper/blob/main/ecr-login/api/client.go#L36

This seems to be the issue.

[paulcdejean]

awslabs/amazon-ecr-credential-helper#1054

[mselim00]

yes image pulls should fallback to unauthenticated if authentication fails. by default, we configure kubelet to use the ecr-credential-provider (not the amazon-ecr-credential-helper)

there's a difference in interface between the two, kubelet invokes the credential provider with a CredentialProviderRequest. That would be an invalid input for the amazon-ecr-credential-helper, which implements the docker credential helper interface and (I think) expects only the image URI as the input, so it makes sense that the custom provider set up in this case is failing

The ecr-credential-provider is currently limited by the same assumption as the helper though: https://github.com/kubernetes/cloud-provider-aws/blob/742c08cb45041fa4650199179947714ad21b49b8/cmd/ecr-credential-provider/main.go#L44

I'll work with my team to see what changes must be made to support authenticating for the dualstack endpoint, and also try to reproduce the failure mode you've observed since it would still be unexpected given the unauthenticated fallback.

[paulcdejean]

@mselim00 there's no urgency here, this can wait until Amazon Linux 2026, which I assume is being worked on considering amazon linux 2023 has some key pieces of software that are end of life such as Python 3.9.

[mselim00]

@paulcdejean glad to hear you're not blocked on this! we can certainly still look into supporting this sooner though