Domain name for registry and CAPI endpoint for VMs

[nuhakala]

This PR adds domain name for registry and CAPI endpoint for the VMs, the domain names don't work and are not used in the management cluster or in the host machine, only in the provisioned VMs. This is done utilizing libvirt dnsmasq instance in the external network. Because the dnsmasq instance is tied to the libvirt network, it should not be used by others than the VMs.

Why we need this?

In IPv6 only setup there were problems with pulling images from IPv6 registry. The container runtimes were not able to decode a hostname containing IPv6 address properly. In addition, the provisioned VMs were not able to pull K8s images from internet. This PR aims to solve those problems.

In addition, also encoding the CAPI endpoint as plain IPv6 address had some problems, hence also that was replaced with domain name.

See #1567 for context

Pull-through cache

This PR adds a new registry container. The new registry is a pull-through cache for the VMs and the VMs will pull the K8s images through that registry. It serves two purposes:

1. In IPv6 only setup pulling images from internet would fail because our environment does not support native IPv6. To solve this, we want to store the K8s images locally. Pulling and pushing them by hand into local registry would be possible, but then we would need to make sure that the registry contains the correct versions of images. Pull-through cache helps to avoid version mismatches in future.
2. It should reduce the outbound network traffic for provisioning VMs because the images are cached locally.

It is not possible to push images to pull-through cache. Hence it is not possible to reuse the existing registry for this purpose.

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign elfosardo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nuhakala]

/test metal3-centos-e2e-integration-test-release-1-12 metal3-dev-env-integration-test-ubuntu-main

[nuhakala]

/test metal3-centos-e2e-integration-test-release-1-12

[Copilot]

Pull request overview

Adds VM-scoped DNS names (via libvirt dnsmasq on the external network) and introduces a dedicated pull-through cache registry for Kubernetes images, primarily to make IPv6-only provisioning more reliable (avoiding IPv6-literal parsing issues and improving image pull reliability).

Changes:

• Adds libvirt dnsmasq host records for a registry domain and the target cluster API endpoint domain within the external VM network.
• Introduces a second local registry container as a pull-through cache for registry.k8s.io and updates kubeadm templates to pull Kubernetes images through it.
• Updates test/provisioning templates to use domain-based registry references and adjusts API endpoint usage in several places.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
vm-setup/roles/common/defaults/main.yml	Adds external-network dnsmasq extra host records (registry + CAPI endpoint) and wires them into libvirt network DNS config.
vars.md	Documents REGISTRY_IP.
tests/roles/run_tests/vars/main.yml	Switches registry variables used in templates to domain-based REGISTRY and adds K8S_REGISTRY.
tests/roles/run_tests/templates/release-1.12/cluster-template-workers-kubeadm-config-ubuntu.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.12/cluster-template-workers-kubeadm-config-leap.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.12/cluster-template-workers-kubeadm-config-centos.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.12/cluster-template-controlplane-kubeadm-config-ubuntu.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.12/cluster-template-controlplane-kubeadm-config-leap.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.12/cluster-template-controlplane-kubeadm-config-centos.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-workers-kubeadm-config-ubuntu.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-workers-kubeadm-config-leap.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-workers-kubeadm-config-centos.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-controlplane-kubeadm-config-ubuntu.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-controlplane-kubeadm-config-leap.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.11/cluster-template-controlplane-kubeadm-config-centos.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.10/cluster-template-workers-kubeadm-config-ubuntu.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.10/cluster-template-workers-kubeadm-config-centos.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.10/cluster-template-controlplane-kubeadm-config-ubuntu.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/release-1.10/cluster-template-controlplane-kubeadm-config-centos.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/metal3datatemplate-template.yaml	Switches node DNS servers to use the external-network gateway (libvirt dnsmasq) instead of public DNS.
tests/roles/run_tests/templates/main/cluster-template-workers.yaml	Sets kubeadm imageRepository (and CoreDNS repo) for workers to the pull-through cache registry.
tests/roles/run_tests/templates/main/cluster-template-workers-kubeadm-config-ubuntu.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-workers-kubeadm-config-leap.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-workers-kubeadm-config-centos.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-controlplane.yaml	Sets kubeadm imageRepository (and CoreDNS repo) for control plane to the pull-through cache registry.
tests/roles/run_tests/templates/main/cluster-template-controlplane-kubeadm-config-ubuntu.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-controlplane-kubeadm-config-leap.yaml	Adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-controlplane-kubeadm-config-centos.yaml	Uses API endpoint IP for keepalived and adds K8S_REGISTRY to insecure registries.
tests/roles/run_tests/templates/main/cluster-template-cluster.yaml	Switches Cluster API controlPlaneEndpoint.host to use CLUSTER_APIENDPOINT_IP.
lib/network.sh	Introduces REGISTRY_IP, hard-sets registry/CAPI endpoint domain variables, and defines pull-through cache port.
03_launch_mgmt_cluster.sh	Switches component image references and KinD mirror config to use a domain-based registry host.
02_configure_host.sh	Adds creation/start of a new k8sregistry pull-through cache container.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[nuhakala]

I made the host hardcoded on purpose. The hostname is not used anywhere else than in the external network where the VMs look for domain names at the libvirt DNS server. The name of the domain is irrelevant, as long as it is not mixed to other domains. Not being able to configure this makes the env slightly simpler for the user.

[nuhakala]

See above comment.

[nuhakala]

The new variables were left undocumented because they are not configurable at the moment.

[nuhakala]

/test metal3-centos-e2e-integration-test-release-1-12 metal3-dev-env-integration-test-ubuntu-main

[nuhakala]

This PR has conflicts but could you still take a look so we get this one rolling as well @Rozzii @tuminoid @Sunnatillo
/cc @terror96

[nuhakala]

/test metal3-centos-e2e-integration-test-release-1-13 metal3-dev-env-integration-test-ubuntu-main

[lentzi90]

/cc

[nuhakala]

/test metal3-dev-env-integration-test-ubuntu-main

[metal3-io-bot]

@nuhakala: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
metal3-centos-e2e-integration-test-release-1-12	43a80db	link	true	/test metal3-centos-e2e-integration-test-release-1-12
metal3-centos-e2e-integration-test-release-1-13	be2642c	link	true	/test metal3-centos-e2e-integration-test-release-1-13
metal3-dev-env-integration-test-ubuntu-main	be2642c	link	true	/test metal3-dev-env-integration-test-ubuntu-main

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[metal3-io-bot]

@nuhakala: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
metal3-centos-e2e-integration-test-release-1-12	43a80db	link	true	/test metal3-centos-e2e-integration-test-release-1-12
metal3-centos-e2e-integration-test-release-1-13	be2642c	link	true	/test metal3-centos-e2e-integration-test-release-1-13
metal3-dev-env-integration-test-ubuntu-main	be2642c	link	true	/test metal3-dev-env-integration-test-ubuntu-main

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[nuhakala]

This is working in dev env, but most likely due to config mismatch it does not work in e2e tests. I don't have time to work more on this right now so will turn this into draft PR.