Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

AruPersia · 2024-09-30T18:07:03Z

AruPersia
Sep 30, 2024

Hi Talos Community,

I’m running a Talos Linux cluster with KubePrism instead of kube-proxy, and I'm having trouble accessing a NodePort (30443) service (NGINX Ingress Controller) via the Virtual IP (VIP). For over 3 months, accessing the NodePort via the VIP (10.10.30.100:30443) worked fine. However, after an outage on one of the nodes, I’m now getting a Connection refused error when using the VIP.

❯ nc -vz 10.10.30.100 30443
nc: connectx to 10.10.30.100 port 30443 (tcp) failed: Connection refused
❯ nc -vz 10.10.30.101 30443
Connection to 10.10.30.101 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 30443
Connection to 10.10.30.102 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 30443
Connection to 10.10.30.103 port 30443 [tcp/*] succeeded!

Direct access to the NodePort on individual node IPs still works.

Any ideas on how to resolve this?

Thanks!

Nodes

❯ kubectl get nodes -o wide
NAME                       STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
xxxxxxx-controlplane-101   Ready    control-plane   87d   v1.30.1   10.10.30.101   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-controlplane-102   Ready    control-plane   87d   v1.30.1   10.10.30.102   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-controlplane-103   Ready    control-plane   87d   v1.30.1   10.10.30.103   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-201         Ready    <none>          87d   v1.30.1   10.10.30.201   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-202         Ready    <none>          87d   v1.30.1   10.10.30.202   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22
xxxxxxx-worker-203         Ready    <none>          87d   v1.30.1   10.10.30.203   <none>        Talos (v1.7.7)   6.6.52-talos     containerd://1.7.22

Nginx Ingress Controller

❯ kubectl -n ingress-nginx-core get svc
NAME                                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-core-external-controller             NodePort    10.107.245.120   <none>        80:30080/TCP,443:30443/TCP   46h
ingress-nginx-core-external-controller-admission   ClusterIP   10.111.184.244   <none>        443/TCP                      46h
ingress-nginx-core-internal-controller             NodePort    10.105.180.252   <none>        80:31080/TCP,443:31443/TCP   46h
ingress-nginx-core-internal-controller-admission   ClusterIP   10.108.38.128    <none>        443/TCP                      46h

Members

❯ talosctl -n 10.10.30.100 get members
NODE           NAMESPACE   TYPE     ID                         VERSION   HOSTNAME                   MACHINE TYPE   OS               ADDRESSES
10.10.30.100   cluster     Member   xxxxxxx-controlplane-101   3         xxxxxxx-controlplane-101   controlplane   Talos (v1.7.7)   ["10.10.30.101"]
10.10.30.100   cluster     Member   xxxxxxx-controlplane-102   3         xxxxxxx-controlplane-102   controlplane   Talos (v1.7.7)   ["10.10.30.102"]
10.10.30.100   cluster     Member   xxxxxxx-controlplane-103   3         xxxxxxx-controlplane-103   controlplane   Talos (v1.7.7)   ["10.10.30.100","10.10.30.103"]
10.10.30.100   cluster     Member   xxxxxxx-worker-201         4         xxxxxxx-worker-201         worker         Talos (v1.7.7)   ["10.10.30.201"]
10.10.30.100   cluster     Member   xxxxxxx-worker-202         4         xxxxxxx-worker-202         worker         Talos (v1.7.7)   ["10.10.30.202"]
10.10.30.100   cluster     Member   xxxxxxx-worker-203         2         xxxxxxx-worker-203         worker         Talos (v1.7.7)   ["10.10.30.203"]

❯ talosctl -n 10.10.30.100 etcd members
NODE           ID                 HOSTNAME                   PEER URLS                   CLIENT URLS                 LEARNER
10.10.30.100   489c0d3d96d436ce   xxxxxxx-controlplane-103   https://10.10.30.103:2380   https://10.10.30.103:2379   false
10.10.30.100   5e0abd023e56c559   xxxxxxx-controlplane-101   https://10.10.30.101:2380   https://10.10.30.101:2379   false
10.10.30.100   821e86300dcfa2c1   xxxxxxx-controlplane-102   https://10.10.30.102:2380   https://10.10.30.102:2379   false

Extensions

❯ talosctl -n 10.10.30.101,10.10.30.102,10.10.30.103 get extensions
NODE           NAMESPACE   TYPE              ID   VERSION   NAME               VERSION
10.10.30.101   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.101   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.101   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.101   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b
10.10.30.102   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.102   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.102   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.102   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b
10.10.30.103   runtime     ExtensionStatus   0    1         iscsi-tools        v0.1.4
10.10.30.103   runtime     ExtensionStatus   1    1         qemu-guest-agent   8.2.2
10.10.30.103   runtime     ExtensionStatus   2    1         util-linux-tools   2.39.3
10.10.30.103   runtime     ExtensionStatus   3    1         schematic          88d1f7a5c4f1d3aba7df787c448c1d3d008ed29cfb34af53fa0df4336a56040b

Ping Test

❯ ping -c 1 10.10.30.100
PING 10.10.30.100 (10.10.30.100): 56 data bytes
64 bytes from 10.10.30.100: icmp_seq=0 ttl=63 time=4.705 ms

--- 10.10.30.100 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.705/4.705/4.705/0.000 ms

❯ ping -c 1 10.10.30.101
PING 10.10.30.101 (10.10.30.101): 56 data bytes
64 bytes from 10.10.30.101: icmp_seq=0 ttl=63 time=3.499 ms

--- 10.10.30.101 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.499/3.499/3.499/0.000 ms

❯ ping -c 1 10.10.30.102
PING 10.10.30.102 (10.10.30.102): 56 data bytes
64 bytes from 10.10.30.102: icmp_seq=0 ttl=63 time=5.076 ms

--- 10.10.30.102 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.076/5.076/5.076/nan ms

❯ ping -c 1 10.10.30.103
PING 10.10.30.103 (10.10.30.103): 56 data bytes
64 bytes from 10.10.30.103: icmp_seq=0 ttl=63 time=4.661 ms

--- 10.10.30.103 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.661/4.661/4.661/nan ms

Connectivity Test on Port 30443

❯ nc -vz 10.10.30.100 30443
nc: connectx to 10.10.30.100 port 30443 (tcp) failed: Connection refused
❯ nc -vz 10.10.30.101 30443
Connection to 10.10.30.101 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 30443
Connection to 10.10.30.102 port 30443 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 30443
Connection to 10.10.30.103 port 30443 [tcp/*] succeeded!

Connectivity Test on Port 6443

❯ nc -vz 10.10.30.100 6443
Connection to 10.10.30.100 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.101 6443
Connection to 10.10.30.101 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.102 6443
Connection to 10.10.30.102 port 6443 [tcp/sun-sr-https] succeeded!
❯ nc -vz 10.10.30.103 6443
Connection to 10.10.30.103 port 6443 [tcp/sun-sr-https] succeeded!

Connectivity Test on Port 50000

❯ nc -vz 10.10.30.100 50000
Connection to 10.10.30.100 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.101 50000
Connection to 10.10.30.101 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 50000
Connection to 10.10.30.102 port 50000 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 50000
Connection to 10.10.30.103 port 50000 [tcp/*] succeeded!

Connectivity Test on Port 50001

❯ nc -vz 10.10.30.100 50001
Connection to 10.10.30.100 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.101 50001
Connection to 10.10.30.101 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.102 50001
Connection to 10.10.30.102 port 50001 [tcp/*] succeeded!
❯ nc -vz 10.10.30.103 50001
Connection to 10.10.30.103 port 50001 [tcp/*] succeeded!

Addresses

❯ talosctl -n 10.10.30.101 get addresses
NODE           NAMESPACE   TYPE            ID                                             VERSION   ADDRESS                        LINK
10.10.30.101   network     AddressStatus   cilium_host/10.244.3.193/32                    1         10.244.3.193/32                cilium_host
10.10.30.101   network     AddressStatus   cilium_host/fe80::8da:88ff:feda:abf0/64        1         fe80::8da:88ff:feda:abf0/64    cilium_host
10.10.30.101   network     AddressStatus   cilium_net/fe80::e0f0:a8ff:fe34:94f9/64        2         fe80::e0f0:a8ff:fe34:94f9/64   cilium_net
10.10.30.101   network     AddressStatus   cilium_vxlan/fe80::b8c3:47ff:fe0d:3382/64      2         fe80::b8c3:47ff:fe0d:3382/64   cilium_vxlan
10.10.30.101   network     AddressStatus   eth0/10.10.30.100/32                           1         10.10.30.100/32                eth0
10.10.30.101   network     AddressStatus   eth0/10.10.30.101/24                           1         10.10.30.101/24                eth0
10.10.30.101   network     AddressStatus   eth0/fe80::8489:eaff:fe6d:7cbf/64              2         fe80::8489:eaff:fe6d:7cbf/64   eth0
10.10.30.101   network     AddressStatus   lo/127.0.0.1/8                                 1         127.0.0.1/8                    lo
10.10.30.101   network     AddressStatus   lo/::1/128                                     1         ::1/128                        lo
10.10.30.101   network     AddressStatus   lxc0dccc825abe1/fe80::286d:a1ff:fef3:20f4/64   2         fe80::286d:a1ff:fef3:20f4/64   lxc0dccc825abe1
10.10.30.101   network     AddressStatus   lxc_health/fe80::7834:66ff:fe9c:b48b/64        2         fe80::7834:66ff:fe9c:b48b/64   lxc_health
10.10.30.101   network     AddressStatus   lxcd07f60af1a96/fe80::6c6e:e8ff:fe9a:123c/64   2         fe80::6c6e:e8ff:fe9a:123c/64   lxcd07f60af1a96

❯ talosctl -n 10.10.30.102 get addresses
NODE           NAMESPACE   TYPE            ID                                             VERSION   ADDRESS                        LINK
10.10.30.102   network     AddressStatus   cilium_host/10.244.4.110/32                    1         10.244.4.110/32                cilium_host
10.10.30.102   network     AddressStatus   cilium_host/fe80::7897:51ff:fea8:2e7d/64       2         fe80::7897:51ff:fea8:2e7d/64   cilium_host
10.10.30.102   network     AddressStatus   cilium_net/fe80::50db:e3ff:fe75:fd67/64        2         fe80::50db:e3ff:fe75:fd67/64   cilium_net
10.10.30.102   network     AddressStatus   cilium_vxlan/fe80::64a4:53ff:fe4a:89e0/64      2         fe80::64a4:53ff:fe4a:89e0/64   cilium_vxlan
10.10.30.102   network     AddressStatus   eth0/10.10.30.102/24                           1         10.10.30.102/24                eth0
10.10.30.102   network     AddressStatus   eth0/fe80::64ce:f8ff:fe3f:8699/64              2         fe80::64ce:f8ff:fe3f:8699/64   eth0
10.10.30.102   network     AddressStatus   lo/127.0.0.1/8                                 1         127.0.0.1/8                    lo
10.10.30.102   network     AddressStatus   lo/::1/128                                     1         ::1/128                        lo
10.10.30.102   network     AddressStatus   lxc07e17807ca04/fe80::3082:f7ff:feeb:8b8/64    2         fe80::3082:f7ff:feeb:8b8/64    lxc07e17807ca04
10.10.30.102   network     AddressStatus   lxc7f5d9b65da86/fe80::e0f3:b7ff:fe6e:d75b/64   2         fe80::e0f3:b7ff:fe6e:d75b/64   lxc7f5d9b65da86
10.10.30.102   network     AddressStatus   lxc_health/fe80::58c0:84ff:fed1:bd1b/64        2         fe80::58c0:84ff:fed1:bd1b/64   lxc_health

❯ talosctl -n 10.10.30.103 get addresses
NODE           NAMESPACE   TYPE            ID                                          VERSION   ADDRESS                        LINK
10.10.30.103   network     AddressStatus   cilium_host/10.244.1.110/32                 1         10.244.1.110/32                cilium_host
10.10.30.103   network     AddressStatus   cilium_host/fe80::10af:beff:fe47:4ae/64     1         fe80::10af:beff:fe47:4ae/64    cilium_host
10.10.30.103   network     AddressStatus   cilium_net/fe80::700e:d9ff:fe33:a92d/64     2         fe80::700e:d9ff:fe33:a92d/64   cilium_net
10.10.30.103   network     AddressStatus   cilium_vxlan/fe80::44ea:adff:fe1e:31e0/64   2         fe80::44ea:adff:fe1e:31e0/64   cilium_vxlan
10.10.30.103   network     AddressStatus   eth0/10.10.30.103/24                        1         10.10.30.103/24                eth0
10.10.30.103   network     AddressStatus   eth0/fe80::3050:31ff:fecd:d95b/64           2         fe80::3050:31ff:fecd:d95b/64   eth0
10.10.30.103   network     AddressStatus   lo/127.0.0.1/8                              1         127.0.0.1/8                    lo
10.10.30.103   network     AddressStatus   lo/::1/128                                  1         ::1/128                        lo
10.10.30.103   network     AddressStatus   lxc_health/fe80::4cd8:88ff:fe5c:858e/64     2         fe80::4cd8:88ff:fe5c:858e/64   lxc_health

Health

❯ talosctl -n 10.10.30.101 health
discovered nodes: ["10.10.30.201" "10.10.30.202" "10.10.30.203" "10.10.30.100" "10.10.30.102" "10.10.30.103"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

smira · 2024-09-30T18:12:20Z

smira
Sep 30, 2024
Maintainer

First of all, Talos VIP is not supposed to be used for applications, but only for controlplane access on port 6443. Every other use is possible, but not supported.

In your output, the address .100 is assigned to the first machine, and that's all what Talos should do, it doesn't manage anything beyond that.

As a wild guess, you regenerated machine config for Talos v1.8, and you enabled new feature: https://www.talos.dev/v1.8/introduction/what-is-new/#default-node-labels

0 replies

maitredede · 2025-12-13T14:49:06Z

maitredede
Dec 13, 2025

Hello, I have a similar use case with same problem. I am using Talos

I have 4 nodes, 3 control-plane, 1 worker (will add more later). I use the vip for controlplanes. I have a basic firewall (1 exposed port redirects to one ip/port). I have deployed nginx ingress as nodeport with externalTrafficPolicy: Local to keep source ip address, but requires the pod to be running on the node.

Since the vip is on control-planes, I deployed nginx ingress as daemonset running on control-plane pods, and service configured as nodeport. When I curl the node ip/nodePort, it works. When I curl the vip/nodePort, it fails.

@smira wrote that Talos VIP is not supposed to be used by applications. What would be the correct way ? Is there a combination of components that could work to expose a single vip shared by nodes having a running nginx ingress pod ?

I made a test with kube-vip. I have a second VIP, shared between two nodes (the worker and one control-plane). I updated nginx ingress installation to be deployed to the node having the vip (using kube-vip label). Even in this case, a curl to node ip/nodePort works, but using the vip does not.

It looks like when a nodePort is created on the node, it is bound to only node ip. It is not "not bound" (like opening the port for all network cards). Is there a way to configure this or a workaround ?

2 replies

smira Dec 15, 2025
Maintainer

This is not a Talos issue, please see Kubernetes documentation on kube-proxy and nftables backend. By default, NodePort services are bound to primary Node IP, this is by design, but you can change that.

With Talos, you need to modify .cluster.proxy section of the machine configuration, adding necessary extraArgs.

Once you're done, talosctl upgrade-k8s --to <same-version-as-installed> should update the kube-proxy deployed manifest in Kubernetes.

maitredede Dec 16, 2025

Thanks for your help, it is more clear for me. I will use the internal vip feature only for control-plane.

Since it is my learing homelab, I have installed metallb and configured an other "vip" for the ingress. It works fine for the use cases I have.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments 2 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Issue with Accessing NodePort via Virtual IP (VIP) on Talos Linux Cluster #9405

Uh oh!

AruPersia Sep 30, 2024

Nodes

Nginx Ingress Controller

Members

Extensions

Ping Test

Connectivity Test on Port 30443

Connectivity Test on Port 6443

Connectivity Test on Port 50000

Connectivity Test on Port 50001

Addresses

Health

Replies: 2 comments · 2 replies

Uh oh!

smira Sep 30, 2024 Maintainer

Uh oh!

maitredede Dec 13, 2025

Uh oh!

smira Dec 15, 2025 Maintainer

Uh oh!

maitredede Dec 16, 2025

AruPersia
Sep 30, 2024

Replies: 2 comments 2 replies

smira
Sep 30, 2024
Maintainer

maitredede
Dec 13, 2025

smira Dec 15, 2025
Maintainer