Skip to content

[BUG] In non-primary-cni mode, primary pod interface of VPC NAT gateway pod attaches to internal network instead of pod network #6632

Open
#6633
Open
[BUG] In non-primary-cni mode, primary pod interface of VPC NAT gateway pod attaches to internal network instead of pod network#6632
#6633
Labels
bugSomething isn't workingvpc
@rrajendran17

Description

@rrajendran17
rrajendran17
opened on Apr 15, 2026

Kube-OVN Version

v1.15.4

Kubernetes Version

v1.35.2

Operation-system/Kernel Version

SUSE Linux 6.12.0-160000.27-default

Description

Ping from VM to external 8.8.8.8 fails when using kubeovn as secondary cni on custom subnet/vpc as the vpc nat gateway pod created has the default pod interface eth0 same ip as internal tenant network and a route is created for internal network via both eth0 and net1,due to which the traffic is affected/not forwarded.

Steps To Reproduce

1.Create a network attachment definition (tenant or internal network)

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
  labels:
    network.harvesterhci.io/ready: "true"
    network.harvesterhci.io/type: OverlayNetwork
  name: vswitchinternal
  namespace: default
spec:
  config: '{"cniVersion":"0.3.1","name":"vswitchinternal","type":"kube-ovn","server_socket":
    "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchinternal.default.ovn"}'

2.Create a network attachment definition (external network)

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
  labels:
    network.harvesterhci.io/ready: "true"
    network.harvesterhci.io/type: OverlayNetwork
  name: vswitchexternal
  namespace: kube-system
spec:
  config: '{"cniVersion":"0.3.1","name":"vswitchexternal","master": "eno50","type":"kube-ovn","server_socket":
    "/run/openvswitch/kube-ovn-daemon.sock", "provider": "vswitchexternal.kube-system.ovn"}'

3.create a subnet using the internal or tenant network in custom vpc named "commonvpc"
(create a vpc called 'commonvpc' from UI before this)

apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: subnetinternal
spec:
  cidrBlock: 172.20.10.0/24
  default: false
  enableLb: true
  excludeIps:
  - 172.20.10.1
  gateway: 172.20.10.1
  gatewayNode: ""
  natOutgoing: true
  private: false
  protocol: IPv4
  provider: vswitchinternal.default.ovn
  vpc: commonvpc
Create a subnet using the external network
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
  name: subnetexternal
spec:
  cidrBlock: 10.115.8.0/21
  default: false
  enableLb: true
  excludeIps:
  - 10.115.15.254
  gateway: 10.115.15.254
  gatewayNode: ""
  gatewayType: distributed
  natOutgoing: true
  private: false
  protocol: IPv4
  provider: vswitchexternal.kube-system.ovn
  vpc: ovn-cluster

5.Create the vpc nat gateway config

kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
  annotations:
        k8s.v1.cni.cncf.io/networks: default/vswitchinternal
  name: gw1
spec:
  vpc: commonvpc
  subnet: subnetinternal
  lanIp: 172.20.10.254
  externalSubnets:
    - subnetexternal

6.Verify if a new vpcnatgw statefulset and a pod created

kubectl describe pod vpc-nat-gw-gw1-0 -n kube-system 
Name:             vpc-nat-gw-gw1-0
Namespace:        kube-system
Priority:         0
Service Account:  default
Node:             n1-hp50/10.115.48.203
Start Time:       Wed, 15 Apr 2026 02:47:39 +0000
Labels:           app=vpc-nat-gw-gw1
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=vpc-nat-gw-gw1-78464c8b78
                  ovn.kubernetes.io/vpc-nat-gw=true
                  statefulset.kubernetes.io/pod-name=vpc-nat-gw-gw1-0
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "default/vswitchinternal",
                        "interface": "eth0",
                        "ips": [
                            "172.20.10.254"
                        ],
                        "mac": "c2:1b:31:99:ce:3d",
                        "default": true,
                        "dns": {},
                        "gateway": [
                            "172.20.10.1"
                        ]
                    },{
                        "name": "default/vswitchinternal",
                        "interface": "net1",
                        "ips": [
                            "172.20.10.254"
                        ],
                        "mac": "c2:1b:31:99:ce:3d",
                        "dns": {}
                    },{
                        "name": "kube-system/vswitchexternal",
                        "interface": "net2",
                        "ips": [
                            "10.115.8.1"
                        ],
                        "mac": "62:8b:76:b5:92:f3",
                        "dns": {},
                        "gateway": [
                            "10.115.15.254"
                        ]
                    }]
                  k8s.v1.cni.cncf.io/networks: default/vswitchinternal, kube-system/vswitchexternal
                  ovn.kubernetes.io/vpc_nat_gw: gw1
                  ovn.kubernetes.io/vpc_nat_gw_init: true
                  v1.multus-cni.io/default-network: default/vswitchinternal
                  vswitchexternal.kube-system.ovn.kubernetes.io/allocated: true
                  vswitchexternal.kube-system.ovn.kubernetes.io/cidr: 10.115.8.0/21
                  vswitchexternal.kube-system.ovn.kubernetes.io/gateway: 10.115.15.254
                  vswitchexternal.kube-system.ovn.kubernetes.io/ip_address: 10.115.8.1
                  vswitchexternal.kube-system.ovn.kubernetes.io/logical_router: ovn-cluster
                  vswitchexternal.kube-system.ovn.kubernetes.io/logical_switch: subnetexternal
                  vswitchexternal.kube-system.ovn.kubernetes.io/mac_address: 62:8b:76:b5:92:f3
                  vswitchexternal.kube-system.ovn.kubernetes.io/pod_nic_type: veth-pair
                  vswitchexternal.kube-system.ovn.kubernetes.io/routed: true
                  vswitchexternal.kube-system.ovn.kubernetes.io/routes: [{"dst":"0.0.0.0/0","gw":"10.115.15.254"}]
                  vswitchinternal.default.ovn.kubernetes.io/allocated: true
                  vswitchinternal.default.ovn.kubernetes.io/cidr: 172.20.10.0/24
                  vswitchinternal.default.ovn.kubernetes.io/gateway: 172.20.10.1
                  vswitchinternal.default.ovn.kubernetes.io/ip_address: 172.20.10.254
                  vswitchinternal.default.ovn.kubernetes.io/logical_router: custom-vpc
                  vswitchinternal.default.ovn.kubernetes.io/logical_switch: subnetinternal
                  vswitchinternal.default.ovn.kubernetes.io/mac_address: c2:1b:31:99:ce:3d
                  vswitchinternal.default.ovn.kubernetes.io/pod_nic_type: veth-pair
                  vswitchinternal.default.ovn.kubernetes.io/routed: true
                  vswitchinternal.default.ovn.kubernetes.io/routes: [{"dst":"10.55.0.0/16","gw":"172.20.10.1"}]
                  vswitchinternal.default.ovn.kubernetes.io/vpc_cidrs: ["172.20.10.0/24"]
Status:           Running
IP:               172.20.10.254
IPs:
  IP:           172.20.10.254
Controlled By:  StatefulSet/vpc-nat-gw-gw1
Containers:
  vpc-nat-gw:
    Container ID:  containerd://cce88945003165ebd8026b71db3c93076d7557c4bfecbe8c8139423f533c2b39
    Image:         docker.io/kubeovn/vpc-nat-gateway:v1.15.4
    Image ID:      docker.io/kubeovn/vpc-nat-gateway@sha256:278001a7b418c362bae4e18bb503ab8d9dd6f0aeb53fb9548eb9a222581cca87
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      infinity
    State:          Running
      Started:      Wed, 15 Apr 2026 02:47:58 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      GATEWAY_V4:  10.115.15.254
      GATEWAY_V6:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-n4jwv (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-n4jwv:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Current Behavior

Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "default/vswitchinternal",
"interface": "eth0",
"ips": [
"172.20.10.254"
],
"mac": "c2:1b:31:99:ce:3d",
"default": true,
"dns": {},
"gateway": [
"172.20.10.1"
]

The eth0 default interface is attached to internal network instead of pod network and internal network route exists via eth0 also.

ip route show
default via 10.115.15.254 dev net2
10.55.0.0/16 via 172.20.10.1 dev net1
10.115.8.0/21 dev net2 proto kernel scope link src 10.115.8.1
172.20.10.0/24 dev eth0 proto kernel scope link src 172.20.10.254
172.20.10.0/24 dev net1 proto kernel scope link src 172.20.10.254

Expected Behavior

eth0 must be attached to default pod network,for example

k8s.v1.cni.cncf.io/network-status:
[{
"name": "k8s-pod-network",
"ips": [
"10.52.0.189"
],
"default": true,
"dns": {}
},{

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingvpc

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions