Kubernetes weave network adapter and network policy

[marcin-kasinski]

Based on https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ I've tried to test network policy on Kubernetes.

I've testes it with script:

while true
do
wget --spider --timeout=2 nginx
sleep 3
done

What you expected to happen?

When execuring my test script I was expected to see:

....
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
...

What happened?

Whene executing test script I see many timeouts

Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
wget: download timed out
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
wget: download timed out
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
wget: download timed out
Connecting to nginx (10.100.122.58:80)
Connecting to nginx (10.100.122.58:80)
wget: download timed out
Connecting to nginx (10.100.122.58:80)
wget: download timed out
Connecting to nginx (10.100.122.58:80)

How to reproduce it?

Prepare nginx , busybox pods, service and network policy based on url I provided.
Then just use my script on busybox labeled pod

Anything else we need to know?

On busybox POD I get

1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
39: eth0 inet 10.32.0.15/12 brd 10.47.255.255 scope global eth0\ valid_lft forever preferred_lft forever

ip route
default via 10.32.0.1 dev eth0
10.32.0.0/12 dev eth0 scope link src 10.32.0.15

In my Kubernetes I have 4 Weave pods.
On one of them when timeout exists I get error:

WARN: 2018/10/19 19:15:32.338824 TCP connection from 10.32.0.1:55588 to 10.42.0.2:80 blocked by Weave NPC.

On other Weave pods there are no errors.

Versions:

kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

uname -a
Linux k8smaster 4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[murali-reddy]

At what stage of the guide https://kubernetes.io/docs/tasks/administer-cluster/declare-network-policy/ you are seeing this? Sorry its not clear if there were any network policies applied or not.

Whether network policies applied or not either all the connection attempts either should succeed or fail, its odd there are consistent intermediate failures.

[marcin-kasinski]

According to pronided url I've applied network policy:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-nginx
spec:
  podSelector:
    matchLabels:
      run: nginx
  ingress:
  - from:
    - podSelector:
        matchLabels:
          access: "true"

This my my main problem .
Network policy applied and somethimes / often connections are blocked.

It is not "always blocked"
It is not "never blocked"
It is "somethimes /often blocked"

[marcin-kasinski]

According to pronided url I've applied network policy:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-nginx
spec:
  podSelector:
    matchLabels:
      run: nginx
  ingress:
  - from:
    - podSelector:
        matchLabels:
          access: "true"

This my my main problem .
Network policy applied and somethimes / often connections are blocked.

It is not "always blocked"
It is not "never blocked"
It is "somethimes /often blocked"

[brb]

@marcin-kasinski Could you provide full logs of the weave-npc container?

[marcin-kasinski]

I've tested it again.
Below you have detailed instruction.

1. Create netpool

echo "" >a.yaml && nano a.yaml && kubectl apply -f a.yaml

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: access-nginx
spec:
  podSelector:
    matchLabels:
      run: nginx
  ingress:
  - from:
    - podSelector:
        matchLabels:
          access: "true"

2. create deployment and service

kubectl run nginx --image=nginx --replicas=2
kubectl expose deployment nginx --port=80

3. Run labeled busybox

kubectl run busybox --rm -ti --labels="access=true" --image=busybox /bin/sh

4. Test with loop inside busybox

while true
do
 wget --spider --timeout=1 nginx
 sleep 2
done

Results:

Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)

It is always blocked

5. I decided to recreate deployment:

kubectl delete deployment nginx
deployment.extensions "nginx" deleted

kubectl run nginx --image=nginx --replicas=2

6. Repeat test with loop inside busybox

while true
do
 wget --spider --timeout=1 nginx
 sleep 2
done

Results:

Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)
wget: download timed out
Connecting to nginx (10.102.123.164:80)

It is better but not good.

7. environment checking within busybox

/ # ip route
default via 10.44.0.0 dev eth0
10.32.0.0/12 dev eth0 scope link  src 10.44.0.20

/ #  ip -4 -o addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
51: eth0    inet 10.44.0.20/12 brd 10.47.255.255 scope global eth0\       valid_lft forever preferred_lft forever

8. Kubernetes environment checking

vagrant@k8smaster:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
vagrant@k8smaster:~$



vagrant@k8smaster:~$ kubectl get po --all-namespaces -o wide| grep 'weave\|nginx\|busybox'
default          busybox-65f7686956-tx7p2                                1/1     Running   0          13m     10.44.0.20     k8snode1    <none>
default          nginx-dbddb74b8-qd5m5                                   1/1     Running   0          8m43s   10.44.0.21     k8snode1    <none>
default          nginx-dbddb74b8-x8qft                                   1/1     Running   0          8m43s   10.39.0.14     k8snode2    <none>
ingress-nginx    nginx-ingress-controller-76f97b74b-9zh5z                1/1     Running   0          33m     10.44.0.12     k8snode1    <none>
ingress-nginx    nginx-ingress-controller-76f97b74b-mqlw8                1/1     Running   0          33m     10.39.0.10     k8snode2    <none>
ingress-nginx    nginx-ingress-controller-76f97b74b-td24l                1/1     Running   0          33m     10.36.0.13     k8snode3    <none>
kube-system      weave-net-2sddl                                         2/2     Running   0          45m     192.168.1.14   k8snode3    <none>
kube-system      weave-net-5k6tm                                         2/2     Running   1          45m     192.168.1.13   k8snode2    <none>
kube-system      weave-net-jvsj2                                         2/2     Running   0          45m     192.168.1.12   k8snode1    <none>
kube-system      weave-net-vtbpt                                         2/2     Running   0          45m     192.168.1.11   k8smaster   <none>

Weave logs placed in: http://itzone.pl/tmp234der/weavelogs.zip

My observations:

I've focused on last test which is not so bad as first one.

In my opinion there is problem with pod: weave-net-5k6tm

In logs of this pod I can see:

...
WARN: 2018/10/24 09:47:49.673265 TCP connection from 10.44.0.0:59604 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:47:50.673265 TCP connection from 10.44.0.0:59604 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:47:52.680357 TCP connection from 10.44.0.0:59628 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:47:52.680479 TCP connection from 10.44.0.0:59628 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:47:57.682264 TCP connection from 10.44.0.0:59672 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:47:58.681258 TCP connection from 10.44.0.0:59672 to 10.39.0.14:80 blocked by Weave NPC.
WARN: 2018/10/24 09:48:02.705445 TCP connection from 10.44.0.0:59714 to 10.39.0.14:80 blocked by Weave NPC. 
...

My questions:

1. Why at the begining connections are always blocked (This is additional question. I haven't discovered it ealier)
2. Why last test ends with "somethimes blocked" ? I was expected to see no timeouts.

[murali-reddy]

@marcin-kasinski thanks for the shared detailed repro steps. However I can not reproduce the issue.

But I observed certain details from the logs.

WARN: 2018/10/24 09:47:49.673265 TCP connection from 10.44.0.0:59604 to 10.39.0.14:80 blocked by Weave NPC

Source IP should have been 10.44.0.20 which is the IP of busybox-65f7686956-tx7p2. 10.44.0.0 is the weave bridge IP. I wonder if kube-proxy is doing the SNAT. By any chance is -masquerade-all=true is set for kube-proxy?

Please try testing again directly with pod IP of nginx and see if it works.

[marcin-kasinski]

I've tested it again from scratch.

When using pod ip not service name it works like a charm.

There are no timeouts.

This is ubuntu on vagrant. This is default kubernetes configuration without my custom changes in kube-proxy or any other system pods.

kube-proxy Daemon set definition:

vagrant@k8smaster:~$ kubectl describe  daemonset -n kube-system kube-proxy
Name:           kube-proxy
Selector:       k8s-app=kube-proxy
Node-Selector:  <none>
Labels:         k8s-app=kube-proxy
Annotations:    <none>
Desired Number of Nodes Scheduled: 4
Current Number of Nodes Scheduled: 4
Number of Nodes Scheduled with Up-to-date Pods: 4
Number of Nodes Scheduled with Available Pods: 4
Number of Nodes Misscheduled: 0
Pods Status:  4 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           k8s-app=kube-proxy
  Annotations:      scheduler.alpha.kubernetes.io/critical-pod:
  Service Account:  kube-proxy
  Containers:
   kube-proxy:
    Image:      k8s.gcr.io/kube-proxy:v1.12.1
    Port:       <none>
    Host Port:  <none>
    Command:
      /usr/local/bin/kube-proxy
      --config=/var/lib/kube-proxy/config.conf
    Environment:  <none>
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/kube-proxy from kube-proxy (rw)
  Volumes:
   kube-proxy:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kube-proxy
    Optional:  false
   xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
   lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules

When logged in to one of kube-proxy pod I can see

cat /var/lib/kube-proxy/config.conf | grep masquerade
  masqueradeAll: false
  masqueradeBit: 14

Entire config:

# cat /var/lib/kube-proxy/config.conf
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
clientConnection:
  acceptContentTypes: ""
  burst: 10
  contentType: application/vnd.kubernetes.protobuf
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 5
clusterCIDR: 10.244.0.0/16
configSyncPeriod: 15m0s
conntrack:
  max: null
  maxPerCore: 32768
  min: 131072
  tcpCloseWaitTimeout: 1h0m0s
  tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
  masqueradeAll: false
  masqueradeBit: 14
  minSyncPeriod: 0s
  syncPeriod: 30s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  syncPeriod: 30s
kind: KubeProxyConfiguration
metricsBindAddress: 127.0.0.1:10249
mode: ""
nodePortAddresses: null
oomScoreAdj: -999
portRange: ""
resourceContainer: /kube-proxy
udpIdleTimeout: 250ms#

[murali-reddy]

There is a mismatch between what kube-proxy thinks as pod CIDR. In the kube-proxy configuration clusterCIDR: 10.244.0.0/16 is specified, so it would assume the pods are in the 10.244.0.0/16. Weave on other hand uses 10.32.0.0/12. Due to this mismatch kube-proxy treats the traffic as outbound so does the masqurade resulting in network policies to fail.

Please see https://github.com/weaveworks/weave/blob/master/site/kubernetes/kube-addon.md#-things-to-watch-out-for

[murali-reddy]

@marcin-kasinski As noted above issue is due to what kube-proxy is configured to think as cluster CIDR. Please reopen if you still see any issue even after giving matching CIDR for both Weave and kube-proxy.