feat(al2): use ecr-credential-provider for public.ecr.aws in 1.27+

[cartermckinnon]

Issue #, if available:

Discussed in #1317

Description of changes:

This allows authenticated public.ecr.aws pulls, to avoid the bandwidth limits for anonymous requests.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.23 / al2	success ✅	success ✅
1.24 / al2	success ✅	success ✅
1.25 / al2	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.29 / al2	success ✅	success ✅
1.30 / al2	success ✅	success ✅

[cartermckinnon]

Folks may not have ECR Public permissions in their node's IAM role, so I need to make sure that kubelet will still attempt a pull if the cred provider returns an error -- I think that's the case IIRC

[sidewinder12s]

Any update on this? @cartermckinnon

[mselim00]

Original assumption seems to be correct, the node still manages to pull the image after the initial error for the ecr-public pull

Feb 10 00:09:50 <NODE_NAME> kubelet[3372]: E0210 00:09:50.270753    3372 plugin.go:235] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image public.ecr.aws/nginx/nginx: exit status 1: I0210 00:09:49.977805 10767 main.go:100] Getting creds for public registry
Feb 10 00:09:50 <NODE_NAME>  kubelet[3372]: E0210 00:09:50.269647   10767 main.go:262] Error running credential provider plugin: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: 9cdb9c8d-4615-48d6-a24a-d5dbd6a77066, api error AccessDeniedException:User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/<MINIMAL_NODE_ROLE>/<INSTANCE_ID> is not authorized to perform: ecr-public:GetAuthorizationToken on resource:* because no identity-based policy allows the ecr-public:GetAuthorizationToken action
Feb 10 00:09:50 <NODE_NAME>  kubelet[3372]: I0210 00:09:50.691738    3372 kubelet.go:2483] "SyncLoop (PLEG): event for pod" pod="default/nginx" event={"ID":"c6436473-30a3-41a4-97ca-6a65add5f409","Type":"ContainerStarted","Data":"0e9d0f92e1c228406dfb12c8b1545c6118a79c6d7982ee2bff99b0ac81ea735c"}
Feb 10 00:09:54 <NODE_NAME> kubelet[3372]: I0210 00:09:54.703589    3372 kubelet.go:2483] "SyncLoop (PLEG): event for pod" pod="default/nginx" event={"ID":"c6436473-30a3-41a4-97ca-6a65add5f409","Type":"ContainerStarted","Data":"eeb7b1f165cfcdd88f4c0856cf69e91fbd852f6bb576e68075de4b1ba2cce625"}

For maintenance purposes, we should consider moving https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L188 here as well, with the new TODO being to move this logic into the config file after 1.26 reaches end of support.

[mselim00]

ci's been updated quite a lot since then so re-running

/ci

[github-actions]

@mselim00 roger that! I've dispatched a workflow. 👍

[github-actions]

@mselim00 the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.25 / al2	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.29 / al2	success ✅	success ✅
1.30 / al2	success ✅	success ✅
1.31 / al2	success ✅	success ✅
1.32 / al2	success ✅	success ✅