NetworkPolicy is blocking traffic to ingress IP

[ameyp]

Environmental Info:
K3s Version: v1.24.1+k3s1 (0581808)

Node(s) CPU architecture, OS, and Version: Linux leader 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: 1 on the control plane, 5 agents

Describe the bug:
I'm running a Pod with two containers: a VPN container (using gluetun) and another container. The VPN is configured to not tunnel traffic headed for CIDR 192.168.99.99/32 and 10.0.0.0/8. I'm applying a network policy that denies all egress other than:

• UDP traffic, VPN port, all IPs
• UDP traffic, DNS port, 10.0.0.0/8
• TCP traffic, HTTPS port, 192.168.99.99/32

My goal is to allow all external traffic via the VPN, and internal traffic only too 192.168.99.99, which is the LoadBalancer IP for Traefik running on my cluster as an ingress controller. With this policy applied, VPN traffic works as expected, but any traffic to any ingress is blocked with connection refused. If I delete the policy, I'm able to reach all ingresses from the pod.

Steps To Reproduce:

• Installed K3s: Using https://github.com/k3s-io/k3s-ansible

Apply the following YAML:

apiVersion: v1
kind: Namespace
metadata:
  name: policy
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
    app.kubernetes.io/name: policy-test
  name: policy
  namespace: policy
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: policy-test
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    to:
    - ipBlock:
        cidr: 10.0.0.0/8
  - ports:
    - port: 443
      protocol: TCP
    to:
    - ipBlock:
        cidr: 0.0.0.0/0
---
apiVersion: v1
kind: Pod
metadata:
  name: policy-test
  namespace: policy
  labels:
    app.kubernetes.io/name: policy-test
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"

Inside the running pod, execute wget https://ingress.hostname.cluster.com where ingress.hostname.cluster.com is any ingress endpoint.

Expected behavior:
Get a valid HTTP response

Actual behavior:

Connecting to ingress.hostname.cluster.com (192.168.99.99:443)
wget: can't connect to remote host (192.168.99.99): Connection refused

Additional context / logs:

[brandond]

Network policies can be pretty difficult to author; we don't really have the resources to assist with their development or troubleshooting but I can at least suggest that you check the policies that apply to the ingress pods (and to the servicelb pod, if you're using that) to confirm that the traffic is allowed through those as well.

[ameyp]

Last mystery solved too: the podSelector by itself only allows traffic to/from selected pods in the same namespace as the policy, so a namespaceSelector and a podSelector must be used together to allow traffic to/from specific pods in other namespaces.

[raghibomar786]

Hey @ameyp, I'm facing a similar issue. Can you share your updated NetworkPolicy that solved the issue?

[ameyp]

  podSelector:
    matchExpressions:
      - key: app.kubernetes.io/name
        operator: In
        values:
          - isolated-app
  egress:
    - ports:
        - protocol: UDP
          port: 51820
      to:
        - ipBlock:
            cidr: 0.0.0.0/0
    - ports:
        - protocol: TCP
          port: 1234
      to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: allowed-app
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: allowed-app
  policyTypes:
    - Egress

[raghibomar786]

Thanks @ameyp . I've a similar policy in place but I get The connection to the server 10.43.0.1:443 was refused error (basically connection to API Server is being refused). Here's my NetworkPolicy :

kind: NetworkPolicy
metadata:
  name: {{ .Chart.Name }}-network-policy
  namespace: {{ .Release.Namespace }}
spec:
  podSelector:
    matchLabels:
      {{- include "my-service.selectorLabels" . | nindent 6 }}
  policyTypes:
    - Ingress
    - Egress
  egress:
  - to: #Allow access to the DNS
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
    ports:
      - protocol: UDP
        port: 53
      - protocol: TCP # Access to kube api server
        port: 443
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
    ports: # ECR access
      - protocol: TCP
        port: 443

 Note: All the egress trafic is allowed from pod, this is because it needs access to api server which doesn't have static ip

[ameyp]

Ah, I don't know if the API server exists as a pod, I always assumed that it sat above pods.