k3s nftables rule of ingress controller bypasses host nftables

[shakibamoshiri]

A new traefik entrypoint (5050/tcp) has been added via helm
The ingress-controller (traefik) should load-balance 5050/tcp to some pods (using ingressRouteTCP CRD) but not always. Sometime there is need to block all incoming ports.

The issue is that when the inet chain of nftable policy set to drop the port 5050/tcp is still accessible

table inet filter {
	chain input {
		type filter hook input priority filter; policy drop;
		iifname "lo" counter accept
		ct state invalid counter drop
		ip saddr 192.168.0.0/16 ip daddr 192.168.0.0/16 counter accept
		ct state invalid counter drop
		ct state established,related counter accept
		ip saddr a.b.c.d counter accept comment "server-nfs"
		ip saddr a.b.c.d counter accept comment "server-204"
		ip6 saddr 2a01:4f8:c012:e50d::1 counter accept
		tcp dport { 22, 80, 443, 989 } accept comment "ssh, http, https, xyz"
	}
}

further checking shows that k3s (or upstream k8s) nftable rules are added to nat table

table ip nat {
	chain KUBE-SERVICES {
		meta l4proto tcp ip daddr 10.43.21.114  tcp dport 443 counter jump KUBE-SVC-CVG3OEGEH7H5P3HQ
		meta l4proto tcp ip daddr public-ip  tcp dport 443 counter jump KUBE-EXT-CVG3OEGEH7H5P3HQ
		meta l4proto tcp ip daddr 10.43.0.10  tcp dport 53 counter jump KUBE-SVC-ERIFXISQEP7F7OF4
		meta l4proto tcp ip daddr 10.43.0.10  tcp dport 9153 counter jump KUBE-SVC-JD5MR3NA4I4DYORP
		meta l4proto tcp ip daddr 10.43.49.240  tcp dport 443 counter jump KUBE-SVC-Z4ANX4WAEWEBLCTM
		meta l4proto tcp ip daddr 10.43.21.114  tcp dport 5050 counter jump KUBE-SVC-EYFF4NBLALWK2FPX
		meta l4proto tcp ip daddr 10.43.21.114  tcp dport 80 counter jump KUBE-SVC-UQMCRMJZLI3FTLDP
		meta l4proto tcp ip daddr public-ip  tcp dport 80 counter jump KUBE-EXT-UQMCRMJZLI3FTLDP
		meta l4proto tcp ip daddr 10.43.0.1  tcp dport 443 counter jump KUBE-SVC-NPX46M4PTMTKRN6Y
		meta l4proto udp ip daddr 10.43.0.10  udp dport 53 counter jump KUBE-SVC-TCOU7JCQXEZGVUNU
                meta l4proto tcp ip daddr public-ip  tcp dport 5050 counter jump KUBE-EXT-EYFF4NBLALWK2FPX 
		 fib daddr type local counter jump KUBE-NODEPORTS
	}
}

the cause are

meta l4proto tcp ip daddr public-ip  tcp dport 443 counter jump KUBE-EXT-CVG3OEGEH7H5P3HQ
meta l4proto tcp ip daddr public-ip  tcp dport 80 counter jump KUBE-EXT-UQMCRMJZLI3FTLDP
meta l4proto tcp ip daddr public-ip  tcp dport 5050 counter jump KUBE-EXT-EYFF4NBLALWK2FPX 

because of these rules , the filter table inet chain is ignored and policy drop has no effect for 80, 443, 5050 or any other entrypoints

I am curious why this way ?
It is possible to block these ports in mangle table but filtering should be done in filter table not mangle

by the way if we delete the nat rule, a few minutes later the rule will be added to nat table

[brandond]

These rules are all managed by kube-proxy. I would probably take your question upstream to the Kubernetes project.

[shakibamoshiri]

I think your document or k8s document should tell which tool should be used as firewall

A few days ago when I added a rule using iptable the k3s-server stopped working and checking the log, showed that that iptable command I added was incompatible , then I removed it

Second thing is if we try iptables-save it gives a warning like this

# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

because of these two, I tried nft tool which seems incompatible with kube-proxy rules

[shakibamoshiri]

I found great detail here KEP-3866: Add an nftables-based kube-proxy backend

kube-proxy uses iptables but in its nftable API mode ¹ -- which means the API being used is nftable but only iptables feature can be used . Kube-proxy has planned to move to nftable ²