Skip to content

Conversation

@zoomoid
Copy link

@zoomoid zoomoid commented Jan 19, 2026

This PR adds the ability to add additional init containers in front of spegel's own configuration init container to perform more host-related initialization via Helm values.

Since GKE is not officially supported by spegel as of right now (https://spegel.dev/docs/getting-started/#gke), we made it work with the host's containerd config by patching the host's config files from another daemonset during node startup. However, this provides inherent possibility of race conditions between spegel and that patch step, where the spegel pod on the node starts before the containerd config is fully patched, sending spegel into a crashloop until the patch is completed.

We want to resolve this by getting rid of the race condition by patching the containerd configuration before spegel starts, thus injecting additional init containers seems to be the easiest solution.

In addition, here's how to make GKE (with ContainerOS) work with spegel:

spec:
  template:
    spec:
      initContainers:
        - name: gke-containerd-patch
          image: busybox:latest
          securityContext:
            privileged: true
          command:
            - nsenter
            - --mount=/proc/1/ns/mnt
            - --
            - /bin/bash
            - -c
            - |-
              set -o errexit
              set -o pipefail
              set -o xtrace
              if grep -q 'config_path' /etc/containerd/config.toml; then
                echo "Containerd drop-in config_path already configured";
                exit 0
              fi
              mkdir -p /etc/containerd/certs.d/docker.io
              # delete old mirror config
              sed -i '/registry\.mirrors\."docker.io"/,+1d' /etc/containerd/config.toml
              sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/' /etc/containerd/config.toml
              # add new mirror config
              cat > "/etc/containerd/certs.d/docker.io/hosts.toml" <<EOF
              server = ["https://mirror.gcr.io", "https://registry-1.docker.io"]
              EOF
              # configure drop-in dir
              cat >> "/etc/containerd/config.toml" <<EOF

              [plugins."io.containerd.grpc.v1.cri".registry]
                config_path = "/etc/containerd/certs.d"
              EOF
              chmod -R o+X+r /etc/containerd/
              # restart containerd to load configuration
              systemctl restart containerd

The rest of the daemonset is omitted for brevity. The customized configuration is persistent during the node's lifecycle.
The script can probably be adjusted to use less privileges and work with a hostPath mount instead of relying on nsenter, but for the time being, this works for us. If you'd like to add this to the "GKE Support" section, feel free to do so.

…osts

the values also contains an example for how to use this to run spegel on GKE's ContainerOS with containerd and systemd

Signed-off-by: Alexander Bartolomey <github@alexanderbartolomey.de>
Signed-off-by: Alexander Bartolomey <github@alexanderbartolomey.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant