IPv6 Support

[xlmnxp]

Summary

Add official IPv6 and dual-stack support for cluster-api-provider-incus to enable Kubernetes clusters with IPv6 nodes, Pods, Services, and control-plane endpoints.

Why is this important?

Incus networks already support IPv6 addressing and routing, and popular CNIs support dual-stack Pod networking. Making IPv6/dual-stack first-class citizens would allow modern network deployments and future-proof clusters.

Are you interested in contributing to this feature?

Yes, I am happy to contribute code and documentation improvements to enable and document IPv6/dual-stack support.

[neoaggelos]

Hey @xlmnxp ! Thanks for opening this issue.

The cluster templates already support POD_CIDR and SERVICE_CIDR (see https://capn.linuxcontainers.org/reference/templates/default.html#cluster-template), so given a network with IPv6 configured, things should work out of the box. I can imagine the default flannel configuration might also need some adjustments to account for IPv6, but you can always ignore it DEPLOY_KUBE_FLANNEL=false and manage your CNI (which is likely what you'd want in a dual-stack cluster).

Can you try it out and come up with a list of issues / shortcomings of the existing manifests? Happy to help address issues, and also add documentation for how to spin up IPv6 / dual-stack clusters.

[xlmnxp]

Hey @xlmnxp ! Thanks for opening this issue.

The cluster templates already support POD_CIDR and SERVICE_CIDR (see https://capn.linuxcontainers.org/reference/templates/default.html#cluster-template), so given a network with IPv6 configured, things should work out of the box. I can imagine the default flannel configuration might also need some adjustments to account for IPv6, but you can always ignore it DEPLOY_KUBE_FLANNEL=false and manage your CNI (which is likely what you'd want in a dual-stack cluster).

Can you try it out and come up with a list of issues / shortcomings of the existing manifests? Happy to help address issues, and also add documentation for how to spin up IPv6 / dual-stack clusters.

I'm facing issue with kubeadm bootstrap pod

E1202 12:18:15.501572       1 cluster_accessor.go:262] "Connect failed" err="error creating HTTP client and mapper: cluster is not reachable: Get \"https://[2a02:9b0:7:5ec6:1266:6aff:fe7a:f6ff]:6443/?timeout=5s\": dial tcp [2a02:9b0:7:5ec6:1266:6aff:fe7a:f6ff]:6443: connect: connection refused" controller="clustercache" controllerGroup="cluster.x-k8s.io" controllerKind="Cluster" Cluster="default/incus-cluster" namespace="default" name="incus-cluster" reconcileID="ecbd0233-528c-4e5d-9f3d-6f6a2819a513"

the cluster can reach IPv6 without issue but the load balancer (LOAD_BALANCER="lxc: {profiles: [default], flavor: c1-m1}") don't listen for port 6443 on IPv6

[xlmnxp]

also I notice there issue with kubelet not starting inside control plane

Dec 02 20:17:27 incus-cluster-lrf6g-w8vm8 kubelet[491]: E1202 20:17:27.826459     491 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"

[neoaggelos]

ah, good catch, you are right. so, the haproxy config template is:

• cluster-api-provider-incus/internal/loadbalancer/config.go

  Lines 73 to 77 in 5ea732d

  	frontend control-plane
  	bind *:{{ .FrontendControlPlanePort }}
  	{{ if .IPv6 -}}
  	bind :::{{ .FrontendControlPlanePort }};
  	{{- end }}

• there is a TODO to handle this in

  cluster-api-provider-incus/internal/loadbalancer/util.go

  Line 32 in 020d583

  // TODO(neoaggelos): care about ipv4 vs ipv6 addresses

  , because we never set .IPv6 in the configData input.

ok, two things you can start with, to test:

• [recommended] can you test with using KUBE_VIP for load-balancer ? should be LOAD_BALANCER=kube-vip: {host: "<ipv6>"} see https://capn.linuxcontainers.org/reference/templates/default.html#load_balancer -- this will not require any haproxy container (so no need to deal with ipv6 related configs at all in that level)

• [harder, but worth a try to iterate quickly] you can set a custom haproxy template in the LXC cluster at .spec.loadBalancer.lxc.customHAProxyConfigTemplate, to enable the IPv6 bind directive (just copy the existing template and enable ipv6 should do it.

kubelets are likely failing because they cannot reach the control plane -- as they also go through the load balancer. i expect i'll have some time during the week to troubleshoot this myself.

thanks!

[xlmnxp]

ah, good catch, you are right. so, the haproxy config template is:

* [cluster-api-provider-incus/internal/loadbalancer/config.go](https://github.com/lxc/cluster-api-provider-incus/blob/5ea732d7afdfee20d224dce54ebb405648091733/internal/loadbalancer/config.go#L73-L77)
      
      
          Lines 73 to 77
        in
        [5ea732d](/lxc/cluster-api-provider-incus/commit/5ea732d7afdfee20d224dce54ebb405648091733)
      
    
    
      
  
          
            
             frontend control-plane 
          
  
          
            
               bind *:{{ .FrontendControlPlanePort }} 
          
  
          
            
               {{ if .IPv6 -}} 
          
  
          
            
               bind :::{{ .FrontendControlPlanePort }}; 
          
  
          
            
               {{- end }}

* there is a TODO to handle this in 
    
      
        [cluster-api-provider-incus/internal/loadbalancer/util.go](https://github.com/lxc/cluster-api-provider-incus/blob/020d58346bbd2694314aaeeb235029c8dadc9b56/internal/loadbalancer/util.go#L32)
      
      
           Line 32
        in
        [020d583](/lxc/cluster-api-provider-incus/commit/020d58346bbd2694314aaeeb235029c8dadc9b56)
      
    
    
      
  
          
            
             // TODO(neoaggelos): care about ipv4 vs ipv6 addresses 
          
      
    
  
  , because we never set .IPv6 in the configData input.

ok, two things you can start with, to test:

* [recommended] can you test with using KUBE_VIP for load-balancer ? should be `LOAD_BALANCER=kube-vip: {host: "<ipv6>"}` see https://capn.linuxcontainers.org/reference/templates/default.html#load_balancer -- this will not require any haproxy container (so no need to deal with ipv6 related configs at all in that level)

* [harder, but worth a try to iterate quickly] you can set a custom haproxy template in the LXC cluster at `.spec.loadBalancer.lxc.customHAProxyConfigTemplate`, to enable the IPv6 bind directive (just copy the existing template and enable ipv6 should do it.

kubelets are likely failing because they cannot reach the control plane -- as they also go through the load balancer. i expect i'll have some time during the week to troubleshoot this myself.

thanks!

for first solution, how I know the container IPv6 before it creating it 😓?
I tried to use Incus host IPv4 and IPv6 didn't work (because missing forwarding)

[neoaggelos]

you don't need the IPv6 of the container, you only to include this fragment in your custom template:

   bind :::{{ .FrontendControlPlanePort }}; 

this will be rendered to:

  bind :::6443;

in the resulting config, and will bind to all addresses. now that i think of it, there might also be another gap, where the cluster load balancer address (.spec.controlPlaneEndpoint) is set to the first IP of the haproxy container, regardless of IPv4 or IPv6.

As such, yeah, ignore the idea of mutating the haproxy configs, and start by trying the kube-vip: { host: "<ipv6>" } instead

[xlmnxp]

you don't need the IPv6 of the container, you only to include this fragment in your custom template:

   bind :::{{ .FrontendControlPlanePort }}; 

this will be rendered to:

  bind :::6443;

in the resulting config, and will bind to all addresses. now that i think of it, there might also be another gap, where the cluster load balancer address (.spec.controlPlaneEndpoint) is set to the first IP of the haproxy container, regardless of IPv4 or IPv6.

As such, yeah, ignore the idea of mutating the haproxy configs, and start by trying the kube-vip: { host: "<ipv6>" } instead

I started with kube-vip:
I see the biggest issue is kubelet use non exists config file as shown in logs above

[neoaggelos]

these logs come from kubelet prior to the cloud-init running. can you grab the following command outputs from the host and share?

cluster-api-provider-incus/test/e2e/shared/log_collector.go

Lines 68 to 95 in 5ea732d

	items := []logitem{
	// version
	{name: "kubelet.version", command: []string{"kubelet", "--version"}},
	{name: "containerd.version", command: []string{"containerd", "--version"}},
	// system stuff
	{name: "system-df-h.log", command: []string{"df", "-h"}},
	{name: "system-uname-a.log", command: []string{"uname", "-a"}},
	{name: "system-ps-fea.log", command: []string{"ps", "-fea"}},
	{name: "system-ip-a.log", command: []string{"ip", "a"}},
	{name: "system-ip-r.log", command: []string{"ip", "r"}},
	{name: "system-iptables-save.log", command: []string{"iptables-save"}},
	{name: "system-lsmod.log", command: []string{"lsmod"}},
	{name: "system-sysctl-a.log", command: []string{"sysctl", "-a"}},
	{name: "system-free-h.log", command: []string{"free", "-h"}},
	// logs
	{name: "cloud-final.log", command: []string{"journalctl", "--no-pager", "-u", "cloud-final"}},
	{name: "kubelet.log", command: []string{"journalctl", "--no-pager", "-u", "kubelet.service"}},
	{name: "containerd.log", command: []string{"journalctl", "--no-pager", "-u", "containerd.service"}},
	{name: "var-log-pods.tar.gz", command: []string{"bash", "-c", "tar cv /var/log/pods | gzip"}},
	// kubernetes configuration
	{name: "etc-containerd.tar.gz", command: []string{"bash", "-c", "tar cv /etc/containerd | gzip"}},
	{name: "etc-kubernetes.tar.gz", command: []string{"bash", "-c", "tar cv /etc/kubernetes | gzip"}},
	{name: "run-kubeadm.tar.gz", command: []string{"bash", "-c", "tar cv /run/kubeadm | gzip"}},
	// container runtime
	{name: "crictl-info.json", command: []string{"crictl", "info"}},
	{name: "crictl-ps-a.log", command: []string{"crictl", "ps", "-a"}},
	{name: "crictl-pods.log", command: []string{"crictl", "pods"}},
	}

these should contain info about what's failing and why

[xlmnxp]

for kubelet I'm getting:

ec 03 19:48:24 incus-cluster-qg8bw-hnvhp systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 9.
Dec 03 19:48:24 incus-cluster-qg8bw-hnvhp systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Dec 03 19:48:24 incus-cluster-qg8bw-hnvhp (kubelet)[508]: kubelet.service: Referenced but unset environment variable evaluates to an empty string: KUBELET_EXTRA_ARGS, KUBELET_KUBEADM_ARGS
Dec 03 19:48:24 incus-cluster-qg8bw-hnvhp kubelet[508]: E1203 19:48:24.558978     508 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Dec 03 19:48:24 incus-cluster-qg8bw-hnvhp systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 19:48:24 incus-cluster-qg8bw-hnvhp systemd[1]: kubelet.service: Failed with result 'exit-code'.

those logs are from inside the container

[neoaggelos]

Yes, these logs come before the cloud-init has bootstrapped the cluster (The kubelet service is already enabled, and just fails until cloud-init has run the kubeadm init step).

So, I'd like the output of these three things (replace incus-cluster-qg8bw-hnvhp with the instance name of the first control plane node):

• incus exec incus-cluster-qg8bw-hnvhp -- journalctl -u cloud-final --no-pager > journactl.log
• incus info incus-cluster-qg8bw-hnvhp > instance.yaml
• incus exec incus-cluster-qg8bw-hnvhp -- tar cv /run/kubeadm | gzip > run-kubeadm.tar.gz

[xlmnxp]

Yes, these logs come before the cloud-init has bootstrapped the cluster (The kubelet service is already enabled, and just fails until cloud-init has run the kubeadm init step).

So, I'd like the output of these three things (replace incus-cluster-qg8bw-hnvhp with the instance name of the first control plane node):

* `incus exec incus-cluster-qg8bw-hnvhp -- journalctl -u cloud-final --no-pager  > journactl.log`

* `incus info incus-cluster-qg8bw-hnvhp > instance.yaml`

* `incus exec incus-cluster-qg8bw-hnvhp -- tar cv /run/kubeadm | gzip > run-kubeadm.tar.gz`

all files attached:-

file instance.yaml:

ame: incus-cluster-77w8k-jq9v6
Description: 
Status: RUNNING
Type: container
Architecture: x86_64
Location: mnode-2
PID: 113264
Created: 2025/12/04 11:07 +03
Last Used: 2025/12/04 11:07 +03
Started: 2025/12/04 11:07 +03

Resources:
  Processes: 21
  CPU usage:
    CPU usage (in seconds): 4
  Memory usage:
    Memory (current): 59.07MiB
  Network usage:
    eth0:
      Type: broadcast
      State: UP
      Host interface: eno1
      MAC address: 10:66:6a:c1:02:76
      MTU: 1500
      Bytes received: 18.50kB
      Bytes sent: 2.18kB
      Packets received: 163
      Packets sent: 19
      IP addresses:
        inet:  192.168.0.143/24 (global)
        inet6: 2a02:9b0:7:5ec6:1266:6aff:fec1:276/64 (global)
        inet6: fe80::1266:6aff:fec1:276/64 (link)
    lo:
      Type: loopback
      State: UP
      MTU: 65536
      Bytes received: 0B
      Bytes sent: 0B
      Packets received: 0
      Packets sent: 0
      IP addresses:
        inet:  127.0.0.1/8 (local)
        inet6: ::1/128 (local)

file journactl.log:
You can view or download it at journactl.log

file run-kubeadm.tar.gz:
You can view or download it at run-kubeadm.tar.gz

I notice there Error about /proc/sys/net/ipv6/conf/default/forwarding inside journactl.log and I can confirm it disabled inside control plane container but enabled in all Incus Cluster hosts

[xlmnxp]

I fixed the issue about IPv6 forwarding and cluster creation succeeded (needed changes are in #176), now I need to fix some issues with Flannel and CoreDNS