Fix CI test for pull-cluster-api-provider-gcp-conformance-ci-artifacts

[damdo]

/kind bug

Fix pull-cluster-api-provider-gcp-conformance-ci-artifacts

As described in #1509 (comment)

[damdo]

cc. @cpanato @justinsb @salasberryfin

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[damdo]

/remove-lifecycle stale

[damdo]

A fix was attempted here: #1556

[damdo]

cc. @justinsb @salasberryfin

[justinsb]

Prow link is https://prow.k8s.io/?job=periodic-cluster-api-provider-gcp-make-conformance-main-ci-artifacts

[justinsb]

I'm a little confused as to how imagebuilder works normally:

We expand containerd.tar (containing ctr) to /usr/local (NOTE: local)

https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/ansible/roles/containerd/tasks/main.yml#L64-L73

We symlink /usr/bin/ctr to point to /usr/local/bin/ctr here:
https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/ansible/roles/kubernetes/tasks/main.yml#L34-L45

But that is called after we invoke url.yaml, which is the task that is blowing up:
https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/ansible/roles/kubernetes/tasks/main.yml#L27-L28

url.yaml is blowing up because it looks for /usr/bin/ctr, but that hasn't been symlinked yet:
https://github.com/kubernetes-sigs/image-builder/blob/main/images/capi/ansible/roles/kubernetes/tasks/url.yml#L85-L87

So I'm guessing that this fails when we hit the condition to import urls.yaml: when: kubernetes_source_type == "http" and kubernetes_cni_source_type == "http"

And indeed:

  if [[ "${USE_CI_ARTIFACTS:-false}" == "true" ]]; then
    cat << EOF > "$(go env GOPATH)/src/sigs.k8s.io/image-builder/images/capi/override.json"
{
  "build_timestamp": "0",
  "kubernetes_source_type": "http",
  "kubernetes_cni_source_type": "http",
  "kubernetes_http_source": "https://dl.k8s.io/ci",
  "kubernetes_series": "v${KUBERNETES_MAJOR_VERSION}.${KUBERNETES_MINOR_VERSION}",
  "kubernetes_semver": "${KUBERNETES_VERSION}"
}
EOF
  else
    cat << EOF > "$(go env GOPATH)/src/sigs.k8s.io/image-builder/images/capi/override.json"
{
  "build_timestamp": "0",
  "kubernetes_series": "v${KUBERNETES_MAJOR_VERSION}.${KUBERNETES_MINOR_VERSION}",
  "kubernetes_semver": "${KUBERNETES_VERSION}",
  "kubernetes_deb_version": "${KUBERNETES_MAJOR_VERSION}.${KUBERNETES_MINOR_VERSION}.${KUBERNETES_PATCH_VERSION}-1.1",
  "kubernetes_rpm_version": "${KUBERNETES_MAJOR_VERSION}.${KUBERNETES_MINOR_VERSION}.${KUBERNETES_PATCH_VERSION}"
}
EOF

So maybe it was working when run locally because USE_CI_ARTIFACTS was not set?

Do we have an example of a different CAPI provider where the equivalent test passes? This would be a nice cross check.

But I think it's an imagebuilder bug, and we should move the "Symlink cri-tools" task "up" (or just symlink it as part of installing containerd, I don't understand the organizational strategy here)

[justinsb]

I did the above by hand, but for fun, I tried asking gemini-cli. It proposed that we set "sysusr_prefix": "/usr/local", which I think would fix this problem but would probably introduce more.

When I asked it why, it said:

✦ This appears to be a configuration mismatch that surfaces when cluster-api-provider-gcp (CAPG) uses image-builder.

  Here is the breakdown of why it is failing:

   1. The Conflict:
       * Where `ctr` is installed: The image-builder Ansible role for containerd defaults to installing binaries in /usr/local/bin (specifically, it uses {{ containerd_prefix | default('/usr/local')
         }}).
       * Where `ctr` is looked for: The image-builder Ansible role for kubernetes uses the variable {{ sysusr_prefix }}/bin/ctr to call the binary.
       * The Mismatch: In image-builder's kubernetes.json configuration file, sysusr_prefix is set to /usr. This causes the script to look for /usr/bin/ctr, which doesn't exist because the binary is
         actually at /usr/local/bin/ctr.

   2. Is it `image-builder` or `cluster-api-provider-gcp`?
       * Technically, it's an issue in `image-builder`'s defaults: The default kubernetes.json configuration (/usr) does not align with the default containerd installation path (/usr/local) for the
         Ubuntu image being built.
       * Practically, it must be fixed in `cluster-api-provider-gcp`: Since CAPG is the consumer driving the build via scripts/ci-conformance.sh, it is responsible for providing the correct
         configuration overrides to ensure the build works. The scripts/ci-conformance.sh script generates an override.json file, and adding "sysusr_prefix": "/usr/local" to that file will resolve the
         path discrepancy.

  This is why I previously recommended updating scripts/ci-conformance.sh to explicitly set sysusr_prefix to /usr/local.

When I nudged it that there should be a symlink:

✦ I have found the "Symlink cri-tools" task in images/capi/ansible/roles/kubernetes/tasks/main.yml:

    1 - name: Symlink cri-tools
    2   ansible.builtin.file:
    3     src: /usr/local/bin/{{ item }}
    4     dest: /usr/bin/{{ item }}
    5     mode: "0755"
    6     state: link
    7     force: true
    8   loop:
    9     - ctr
   10     - crictl
   11     - critest
   12   when: ansible_os_family != "Flatcar"

  This task explicitly creates a symlink from /usr/local/bin/ctr to /usr/bin/ctr.

  However, the "Load Kubernetes images" task (which fails) is located in url.yml, which is imported before main.yml reaches the symlink creation step.

  In images/capi/ansible/roles/kubernetes/tasks/main.yml:

   1 - ansible.builtin.import_tasks: url.yml
   2   when: kubernetes_source_type == "http" and kubernetes_cni_source_type == "http"
   3 
   4 # ...
   5 
   6 - name: Symlink cri-tools
   7   # ...

  The url.yml file is imported before the symlinks are created. This explains why ctr is not found when url.yml executes Load Kubernetes images:

   1 - name: Load Kubernetes images
   2   ansible.builtin.shell: CONTAINERD_NAMESPACE="k8s.io" {{ sysusr_prefix }}/bin/ctr --address={{ containerd_cri_socket }} images import /tmp/{{ item }}
   3   loop: "{{ kubernetes_imgs }}"

  The sysusr_prefix defaults to /usr, so it tries to run /usr/bin/ctr, but the symlink hasn't been created yet.

So ... not bad. Maybe even clearer than my explanation 😂

[salasberryfin]

Thanks for the investigation @justinsb. Unsurprisingly none of the tasks that are called before the symlink is created on other conditions invoke ctr, which may explain why this error has not being caught in other projects.

It looks like "kubernetes_source_type": "http" && "kubernetes_cni_source_type": "http" is a very specific configuration and in fact most image building operations would simply not face this error.

Based on what you shared, I'll open a PR in image-builder to suggest a fix to this.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale