NetPol issue affecting pods in different namespaces across different hosts

[tipruzs]

What you expected to happen?

Create k8s network policy "ALLOW all traffic from a namespace" to share a pods app across the cluster.

What happened?

The connection requests are dropped from weave npc, if the source pod is located on a different host than the target pod.

2019/04/22 10:10:57.664451 TCP connection from 10.46.0.0:32812 to 10.40.0.3:9273 blocked by Weave NPC

How to reproduce it?

• minimal k8s cluster with 1 master and 2 worker nodes
• create deployment and service "nginx1" in namespace "application" with affinity to node1
• create deployment and service "nginx2" in namespace "application" with affinity to node2
• create instant pod in different namespace on any node and try to access the two services

kubectl run test-$RANDOM --rm -it --image=alpine -- sh
apk add --no-cache curl
curl nginx1.application.svc.cluster.local
curl nginx2.application.svc.cluster.local

• import network policy and re-run the tests

Anything else we need to know?

• pharos-cluster 2.3.6 with weave enabled
• service_cidr: 10.96.0.0/12
• pod_network_cidr: 10.32.0.0/12
• kube-proxy

clusterCIDR: 10.32.0.0/12
masqueradeAll: false
mode: iptables

Versions:

$ weave version
2.5.1

$ docker version
18.06.1-ce

$ uname -a
Linux 4.15.0-47-generic #50-Ubuntu SMP

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"archive", BuildDate:"2019-03-01T20:58:08Z", GoVersion:"go1.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Logs:

$ kubectl logs -n kube-system <weave-net-pod> weave

weave.log

Network:

$ ip route
$ ip -4 -o addr
$ sudo iptables-save

iptables.log

[murali-reddy]

Create k8s network policy "ALLOW all traffic from a namespace" to share a pods app across the cluster.

Can you please share an e.g network policy that you used. I understand the part "ALLOW all traffic from a namespace", but not share a pods app across the cluster.. Are you referring to a network policy that use both pod selectors and namespace selector?

[tipruzs]

Sorry for confusion. This is the networkpolicy i used:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-ns-monitoring
  namespace: application
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: nginx
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          app.kubernetes.io/name: monitoring
    ports:
    - protocol: TCP
      port: 80

The connection attempt afterwards is from a pod inside the namespace labeled as "app.kubernetes.io/name=monitoring".

[murali-reddy]

I tried the scenario. I don't see any issue. I am able to access the service exposed by the pods selected by podSelector from any pod in the namespaces selected by namespaceSelector

[tipruzs]

Did you try the connection from different nodes? In my setup it works if source and target are on the same node, but not if the connection is cross-node.
I would like to try this again in a fresh minimal setup. are you able to give me some details about your environment?

[murali-reddy]

Yes, its from different nodes I am able to see its working.

So i created three namespaces n1, n2, n3. Deployed guestbook app in n1 and n2. and applied below network policy.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-n2
  namespace: n1
spec:
  podSelector:
    matchLabels:
      tier: frontend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          namespace: n2 
    ports:
    - protocol: TCP
      port: 80

with below pods running

± k get pods -o wide --all-namespaces -l app=guestbook                                                                                                                                                                                                                                ❯❯❯
NAMESPACE   NAME                        READY     STATUS    RESTARTS   AGE       IP          NODE          NOMINATED NODE   READINESS GATES
n1          frontend-69859f6796-48lpf   1/1       Running   0          13h       10.34.0.3   weave-node2   <none>           <none>
n1          frontend-69859f6796-4z5mz   1/1       Running   0          13h       10.34.0.2   weave-node2   <none>           <none>
n1          frontend-69859f6796-tj45g   1/1       Running   0          13h       10.32.0.5   weave-node1   <none>           <none>
n2          frontend-69859f6796-97d2d   1/1       Running   0          13h       10.32.0.7   weave-node1   <none>           <none>
n2          frontend-69859f6796-gfhb5   1/1       Running   0          13h       10.34.0.5   weave-node2   <none>           <none>
n2          frontend-69859f6796-jhpb6   1/1       Running   0          13h       10.34.0.6   weave-node2   <none>           <none>

I am able to curl frontend-69859f6796-tj45g which is n1 namespace from frontend-69859f6796-gfhb5 which is in n2 namespace

[tipruzs]

thanks, i build a new setup and followed your tests. the only difference is that i connect to a service not to the pod directly. but indeed, it just works. so i need to find out, whats wrong with my prod env.

[murali-reddy]

Oh if you are accessing service then it might not work. NetworkPolices works as expected when there is direct pod-to-pod communication. But pod accessing a services resulting in forwarding traffic to a pod, network policies does not necessarily work, as source IP might get masqueraded. Check (by tcpdump on destination node) if this is the case you are running into.

Its not a implementation issue, rather the semantics of network policies https://groups.google.com/forum/#!topic/kubernetes-sig-network/_KLaN2GKuxM

[tipruzs]

Thanks for clarification. I checked this in my lab env and there seems to be no masquerading, when accessing the service address. The target pod (nginx) reports the correct client adress (checked inside the source pod).

Furthermore i found this in the docs (https://kubernetes.io/docs/concepts/services-networking/service/):

The iptables proxier does not obscure in-cluster source IPs

So i think this should work as long as i go with iptables-mode for kube-proxy.

[tipruzs]

I checked the source ips in prod env without any network policy and found out that the requests are always masqueraded, even if the call goes directly to the target pod. thats quite strange, because both clusters are set up with the same tools/configs.

[tipruzs]

the issue could be solved. the reason was, that there was a stale iptables rules from firewalld, which enforced the masquerading (kontena/pharos-cluster#1302)