Spegel mirrors not used on EKS 1.34 with containerd 2.1 (transfer service bypasses hosts.toml)

[gyanesh-mishra]

Problem

Spegel deploys correctly on EKS 1.34 (AL2023 AMI, containerd 2.1.5) — DHT bootstraps, content is advertised, hosts.toml is written with the correct node IP. But containerd never sends requests to the Spegel mirror. Zero HTTP requests hit the /v2/ endpoint, spegel_mirror_last_success_timestamp_seconds stays at 0.

Root Cause

containerd 2.1 defaults to use_local_image_pull = false, which means CRI image pulls go through the transfer service (io.containerd.transfer.v1.local). The transfer service does not read hosts.toml mirror configuration from config_path. Only the CRI local image pull path honors hosts.toml.

Fix

Setting use_local_image_pull = true in the containerd config forces the CRI plugin to use the local pull path, which reads hosts.toml:

[plugins.'io.containerd.cri.v1.images']
use_local_image_pull = true

After this change, Spegel immediately starts receiving mirror requests (confirmed 9 cache hits on first pull).

Environment

• EKS 1.34, AL2023 AMI
• containerd 2.1.5
• Spegel 0.6.0
• containerdMirrorAdd = true

Suggestion

Document use_local_image_pull = true as a requirement for EKS 1.34+ (or any platform running containerd 2.1+ with the transfer service enabled) in the compatibility/getting started docs.

[phillebaba]

@gyanesh-mishra just FYI I converted the bug issue to a discussion as this is not a bug in Spegel but rather an integration question.

I have been following some of the work that has been done in Containerd in relation to the transfer service work. There was some early issue with the transfer service not picking up the proper mirror configuration.
containerd/containerd#12808
containerd/containerd#12636

I agree that we should document what is the proper configuration required in the docs. Along with that it would be interesting if the Containerd integration tests could also verify that transfer service works with Spegel.

[iciclespider]

@gyanesh-mishra According to @phillebaba second link above containerd/containerd#12636, if you do not configure plugins."io.containerd.cri.v1.images".registry.config_path to be /etc/containerd/certs.d and instead leave in place the default of /etc/containerd/certs.d:/etc/docker/certs.d, it seems mirrors are not used.

Did you change that config_path setting?

[iciclespider]

@phillebaba This is something to be aware of in EKS deployments.

EKS by default sets discard_unpacked_layers = true. This has the affect of forcing containerd to use local image pulling, rather than the transfer service. This is confirmed in the containterd logs:

level=warning msg="Found 'DiscardUnpackedLayers' in CRI config which is incompatible with transfer service (not supported with transfer service). Falling back to local image pull mode."

By setting discard_unpacked_layers = false, there are no longer any configured incompatibilities with the transfer service (as documented here), so this change is also enabling the containterd transfer service. This is confirmed by the containerd logs no longer emitting the above warning. This also explains why it was necessary to also configure the registry mirror config_path with a setting that works with the transfer service.

It might make more sense to instead configure containerd with the following in EKS, to keep containerd configured as close as possible to it's original state:

[plugins."io.containerd.cri.v1.images"]
  discard_unpacked_layers = false
  use_local_image_pull = true

[phillebaba]

Rather than forcing Containerd to not use the transfer service we should get the mirror configuration to work properly with transfer service. I always assumed that it would fallback to the old configuration if nothing was set but it seems I am wrong. Best thing to do is test locally with and without local pull and see what works.