EKS fails to pull images from ECR after upgrading to Knative Serving 1.17.0

[ssagi118]

What version of Knative?

Knative serving v1.17.0 k8S 1.32

Expected Behavior

I have an image in ECR with simple application that exposes REST endpoint.
I use the following .yaml to deploy on EKS

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: dummy-model
  namespace: default
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/min-scale: "1"
    spec:
      containers:
      - image: 111111111111.dkr.ecr.eu-west-1.amazonaws.com/dummy-model:1.5
        ports:
        - containerPort: 4000

This works perfectly with Knative serving v1.16.2 the image is pulled and revision, deployment, pod are created and application is responsive to REST calls.

Actual Behavior

When upgrading to Knative serving v1.17.0 and deploying the same applicatin on the same EKS the output of kn revisions list is:

NAME                SERVICE       TRAFFIC   TAGS   GENERATION   AGE   CONDITIONS   READY   REASON
dummy-model-00001   dummy-model                    1            3s    0 OK / 3     False   ContainerMissing : Unable to fetch image "1111 ...

When looking at the log of the controller (kubectl logs controller-cc7d86698-p648q -n knative-serving)
I see the following:

"severity": "ERROR",
    "timestamp": "2025-02-14T09:17:33.629925989Z",
    "logger": "controller",
    "caller": "controller/controller.go:564",
    "message": "Reconcile error",
    "commit": "6265a8e",
    "knative.dev/pod": "controller-cc7d86698-p648q",
    "knative.dev/controller": "knative.dev.serving.pkg.reconciler.revision.Reconciler",
    "knative.dev/kind": "serving.knative.dev.Revision",
    "knative.dev/traceid": "544e9974-8628-4fd9-a08c-2822eca4a357",
    "knative.dev/key": "default/dummy-model-00001",
    "duration": "253.88µs",
    "error": "Unable to fetch image \"111111111111.dkr.ecr.eu-west-1.amazonaws.com/dummy-model:1.5\": failed to resolve image to digest: HEAD https://111111111111.dkr.ecr.eu-west-1.amazonaws.com/v2/dummy-model/manifests/1.5: unexpected status code 401 Unauthorized (HEAD responses have no body, use GET for details)",
    "stacktrace": "knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:564\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:541\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/controller/controller.go:489"

Steps to Reproduce the Problem

Create an EKS with K8S v1.32 using Hashikorp standard Terraform script.
Create an image with dummy application, push to ECR, deply as Knative serving v1.17.0

[dprotaso]

Someone else reported this but said it also affected 1.16 and earlier

I bumped the library we use to perform the tag to digest resolution - it's in the latest nightly - can you confirm it fixes your issue and I can cherry pick the change back to v1.17.x

https://gcsweb.knative.dev/gcs/knative-nightly/serving/latest

PR with the change is here: #15754

[dprotaso]

/triage accepted

[jgournet]

thank you @dprotaso for this fix !
any chance this could be backported to 1.17.x ?
this would help a lot

[richardvflux]

experimented with this in one of our clusters and it has not alleviated it. However, this may be due to the fact we have a hop count set to 1, we hoped shifting to this nightly would allow this to work but it appears to not do so. How the container is authenticating itself to AWS is still baffling as there is no mechanism to mount a docker config file for the libraries to understand the auth chain, and patching one in does not help.

[dprotaso]

@richardvflux You can try to follow the steps i did in a prior debugging to help isolate the issue

#9477 (comment)

tl;dr under the hood the ecr credential helper is used

https://github.com/awslabs/amazon-ecr-credential-helper?tab=readme-ov-file#aws-credentials

[LCaparelli]

@dprotaso the nightly build fixed it for me. I have installed it via knative-operator, using overrides (sharing to facilitate a workaround for other folks to test):

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
spec:
  registry:
    override:
      controller: >-
        gcr.io/knative-nightly/knative.dev/serving/cmd/controller@sha256:c11f71f486542a2a1fde95e1a22f6f3800881acaae0db9b1643fdb07cf9d5748
  version: 1.17.0

Unclear whether it's relevant, but in our setup (that's running into the same problem: things work fine on 1.16, break on 1.17) we're relying on the node authentication data (instance profiles), and not using a specific AWS IAM Role bound to the controller's SA. It should just work with the credential helper, AFAIK, weird that it stopped working.

PS: missed opportunity to call the nightly repo knightly. Get it? K-nightly? :P

[richardvflux]

We figured out that if we drop the hop count to 1 so that it cannot fall back to the node, we need to make sure the controller has an EKS role associated with it so the appropriate env vars get injected when the controller pods start.

[omartheironman]

is there any update on this? im using IRSA to pull the container images do i need the helper? or is IRSA enough?