[ci.jenkins.io] Docker pull fails with random `ERROR: failed to read expected number of bytes: unexpected EOF`

[dduportal]

Since we migrated ci.jenkins.io to AWS and introduced a transparent Docker registry mirror, we have seen random errors when Docker pull images from the registry.

Symptoms:

• The exact error is ERROR: failed to read expected number of bytes: unexpected EOF
• The error only appears during layer pull: we saw other EOF during extraction or at the end of builds which are totally different problems.
• The datadog logs for the registry source (the Docker "distribution" system deployed under the name hub-mirror in cijenkinsio-agents-2 EKS cluster) shows that the GET requests are responded with HTTP/200 but a body sizer of 0

See the description by @MarkEWaite in jenkinsci/docker#1996

[dduportal]

So, after analysis, it looks like some layers are corrupted in the pull-through registry. Their size is zero even on disk, while the metadata advertise the real size.

If we cleanup the whole disk with a dirty rm -rf and restart the pod, it goes back to normal.

I've tracked down the issue to distribution/distribution#2367 (comment)

[dduportal]

Update:

• Applied fix(ci.jenkins.io-agents-2) disable blob caching as a hack to avoid corrupted zero-sized blobs kubernetes-management#6359
• Removed all cache in the service

Let's see if it helps for #4585 (comment)

[dduportal]

Problem looks gone!