Solving ImagePullBackOff, wrong diff id calculated on extraction (AWS ECR, EKS, Prefect Deploy to ECR for Kubernetes)

[samgreenwood-centric]

TLDR: if you're seeing the errors in the title, try disabling the Docker Desktop setting "Use containerd for pulling and storing images". It's in the General settings area.

Issue experienced

Super simple Prefect setup, just testing out deploying flow changes for our Kubernetes workpool on EKS to use.
flows/
my_flow.py
requirements.txt
prefect.yaml

YAML config was basically a copy of the version in the docs for building Docker images for Kubernetes.

Whenever I ran prefect deploy and selected a flow, it would fail after Step 2 (Step 2/6 : COPY requirements.txt /opt/prefect/flows/requirements.txt) with an error around not having found the image it wanted (despite Step 1 apparently completing a Docker pull of the prefecthq/prefect image).

I could get the deployment to complete by manually running docker pull FROM prefecthq/prefect:3.2.7-python3.12 and specifying the same version Prefect said it was trying to grab.

However, after doing this, even though the deployment process would complete and upload an image to ECR -- whenever EKS tried to pull this image to run a flow, it would fail with the above errors (wrong diff id calculated on extraction).

Worth saying my colleague was able to deploy and run this same toy project without any issues.

Solution

We tried a lot on my machine with installing/re-installing Docker, different package manager venvs, different project layouts for prefect.yaml, etc.

In the end, what seems to have fixed it is unchecking this one option in Docker Desktop, which is on by default now: