NetworkPolicy blocking traffic

[ernoaapa]

Hi,
I'm having problems to apply following, quite simple NetworkPolicy:

---
# Block all ingress traffic by default
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
 name: default-deny-ingress
 namespace: kube-system
spec:
  podSelector: {}
  policyTypes:
  - Ingress
---

# Allow pod-to-pod in same namespace
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: namespace-internal-ingress-allow
  namespace: kube-system
spec:
  podSelector: {}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: kube-system
---

# Allow ingress traffic to coredns
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: coredns-ingress-allow
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      k8s-app: coredns
  ingress:
  # Allow access from any namespace and pod inside Kubernetes, not outside
  - from:
    - namespaceSelector: {}
    - podSelector: {}
    ports:
    - protocol: UDP
      port: 53

What you expected to happen?

Able to connect from any pod in any namespace to the coredns UDP 53.

What happened?

In some cases I was able to connect, and in some other cases not.
Tested nslookup google.com <ip of the coredns pod> following:

• From same host: YES
• From pod in same host and in same namespace: YES
• From pod in same host and different namespace: NO (no logging about blocking)
• From different host: NO (blocked by NPC)
• From pod in different host and in same namespace: NO (blocked by NPC)
• From pod in different host and different namespace: NO (blocked by NPC)

When running from another node, I can see from the weave-npc logs that it's blocking the access.
WARN: 2018/05/17 05:04:08.917079 UDP connection from 10.2.32.0:55996 to 10.2.96.1:53 blocked by Weave NPC.

But if in same host, but in pod which is in different namespace, wasn't able to connect and also did not get any blocked by Weave NPC warning message.

If I remove all the above NetworkPolicies, all above test cases passes.

How to reproduce it?

• setup pretty basic Kubernetes cluster
• apply above rules for kube-system
• exec in some other pod nslookup google.com <ip of the coredns pod>

Versions:

$ weave version
2.2.1
$ docker -v
Docker version 17.12.1-ce, build 7390fc6
$ kubectl version
v1.9.5

[brb]

@ernoaapa Thanks for the issue.

Can you please attach the full weave-npc log from the host running CoreDNS?

[ernoaapa]

@brb sure!
Here's is for example from a different node and the Pod is in a different namespace:

WARN: 2018/05/18 02:43:22.714791 UDP connection from 10.2.32.0:48628 to 10.2.96.1:53 blocked by Weave NPC.
WARN: 2018/05/18 02:43:25.210779 UDP connection from 10.2.32.0:48628 to 10.2.96.1:53 blocked by Weave NPC.
WARN: 2018/05/18 02:43:27.706745 UDP connection from 10.2.32.0:60572 to 10.2.96.1:53 blocked by Weave NPC.
WARN: 2018/05/18 02:43:27.706793 UDP connection from 10.2.32.0:60572 to 10.2.96.1:53 blocked by Weave NPC.
WARN: 2018/05/18 02:43:30.202925 UDP connection from 10.2.32.0:60572 to 10.2.96.1:53 blocked by Weave NPC.
WARN: 2018/05/18 02:43:30.203430 UDP connection from 10.2.32.0:60572 to 10.2.96.1:53 blocked by Weave NPC.

[brb]

@ernoaapa Thanks, but I need the full log.

[brb]

I couldn't reproduce the issue myself. One observation - to make the second policy functional, you need to label the kube-system namespace by kubectl label namespace kube-system name=kube-system.

Closing due to missing logs. Feel free to re-open once you have them.

[brb]

Also, if you use nslookup from BusyBox, please be aware of this bug https://bugs.busybox.net/show_bug.cgi?id=675

[ernoaapa]

Tested with weave-kube:2.5.1 and cannot reproduce this anymore. The namespace-internal-ingress-allow was formated little bit differently (if it make any difference):

# Allow pod-to-pod in same namespace
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: namespace-internal-ingress-allow
  namespace: default
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}