MTU size mismatch among "ip a" , "weave status connection" and "docker logs weave"

[deshui123]

I start weave in cni plugin.
Although I modify WEAVE_MTU to 1464 and the docker logs also show the MTU changed to 1464, "ip a" or "weave status connections" still show the MTU is 1376.
When I reboot VM, "ip a" and "weave status connections" show the MTU is 1464, but "weave status connections" run on other VM which is not reboot still show the MTU is 1376.
My questions:

1. Change MTU before weave launch, why reboot VM is needed?
2. Why MTU mismatch between "weave status connection" and "docker logs weave"
3. why "weave status connection" show diff MTU on diff VMs?
   ==================Before reboot h-mul-worker-edge-02=================
   [root@h-mul-worker-edge-01 ~]$ ip a
   2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc pfifo_fast state UP qlen 1000
   link/ether fa:16:3e:53:04:a0 brd ff:ff:ff:ff:ff:ff
   inet 192.168.10.134/24 brd 192.168.10.255 scope global dynamic eth0
   valid_lft 84172sec preferred_lft 84172sec
   16: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP qlen 1000
   link/ether 2e:90:9f:3f:68:1c brd ff:ff:ff:ff:ff:ff
   inet 192.168.1.64/24 scope global weave
   valid_lft forever preferred_lft forever

[root@h-mul-worker-edge-01 ~]$ weave status connections
-> 192.168.10.132:6783 established encrypted fastdp da:c6:c9:1d:69:6b(h-mul-control-worker-edge-storage-01) encrypted=truemtu=1376
<- 192.168.10.136:55940 established encrypted fastdp f2:c3:6b:d2:79:86(h-mul-control-worker-edge-storage-02) encrypted=truemtu=1376
-> 192.168.10.135:6783 established encrypted fastdp 96:ba:36:60:bc:31(h-mul-worker-edge-02) encrypted=truemtu=1376
<- 192.168.10.133:34488 established encrypted fastdp 9a:b2:24:e2:69:b1(h-mul-control-worker-edge-storage-03) encrypted=truemtu=1376

==============after reboot h-mul-worker-edge-02=============

[root@h-mul-worker-edge-02 ~]$ date;ip a
Tue May 8 04:04:11 UTC 2018
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1550 qdisc pfifo_fast state UP qlen 1000
link/ether fa:16:3e:9b:43:4c brd ff:ff:ff:ff:ff:ff
inet 192.168.10.135/24 brd 192.168.10.255 scope global dynamic eth0
valid_lft 86269sec preferred_lft 86269sec
8: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1464 qdisc noqueue state UP qlen 1000
link/ether 96:ba:36:60:bc:31 brd ff:ff:ff:ff:ff:ff

[root@h-mul-worker-edge-01 ~]$ date;weave status connections
Tue May 8 04:04:15 UTC 2018
-> 192.168.10.132:6783 established encrypted fastdp da:c6:c9:1d:69:6b(h-mul-control-worker-edge-storage-01) encrypted=truemtu=1376
<- 192.168.10.136:55940 established encrypted fastdp f2:c3:6b:d2:79:86(h-mul-control-worker-edge-storage-02) encrypted=truemtu=1376
-> 192.168.10.135:6783 established encrypted fastdp 96:ba:36:60:bc:31(h-mul-worker-edge-02) encrypted=truemtu=1376
<- 192.168.10.133:34488 established encrypted fastdp 9a:b2:24:e2:69:b1(h-mul-control-worker-edge-storage-03) encrypted=truemtu=1376

[root@h-mul-worker-edge-02 ~]$ date;weave status connections
Tue May 8 04:23:09 UTC 2018
<- 192.168.10.133:57192 established encrypted fastdp 9a:b2:24:e2:69:b1(h-mul-control-worker-edge-storage-03) encrypted=truemtu=1464
-> 192.168.10.132:6783 established encrypted fastdp da:c6:c9:1d:69:6b(h-mul-control-worker-edge-storage-01) encrypted=truemtu=1464
-> 192.168.10.136:6783 established encrypted fastdp f2:c3:6b:d2:79:86(h-mul-control-worker-edge-storage-02) encrypted=truemtu=1464
<- 192.168.10.134:55148 established encrypted fastdp 2e:90:9f:3f:68:1c(h-mul-worker-edge-01) encrypted=truemtu= 1464

====================================================================

[root@h-mul-worker-edge-02 ~]$ cat /etc/systemd/system/weave.service
[Unit]
Description=Weave Network
Documentation=http://docs.weave.works/weave/latest_release/
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=20
TimeoutStartSec=20m

EnvironmentFile=-/etc/sysconfig/weave
Environment="CHECKPOINT_DISABLE=1"
Environment="WEAVE_MTU=1464"

ExecStartPre=-/usr/bin/docker rm -f weave
ExecStartPre=/usr/bin/weave launch --no-restart --no-dns --ipalloc-range 192.168.1.0/24 --password $WEAVE_PASSWORD $PEERS
ExecStart=/usr/bin/docker attach weave
ExecStop=/usr/bin/weave stop
ExecStopPost=-/usr/bin/docker stop -t 5 weave

[Install]
WantedBy=multi-user.target

[root@h-mul-worker-edge-01 ~]$ docker logs weave
INFO: 2018/05/08 02:58:59.807484 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/05/08 02:58:59.809011 sleeve ->[192.168.10.132:6783|da:c6:c9:1d:69:6b(h-mul-control-worker-edge-storage-01)]: Effective MTU verified at 1464
INFO: 2018/05/08 02:58:59.816841 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/05/08 02:58:59.818121 sleeve ->[192.168.10.136:6783|f2:c3:6b:d2:79:86(h-mul-control-worker-edge-storage-02)]: Effective MTU verified at 1464
INFO: 2018/05/08 02:58:59.829008 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2018/05/08 02:58:59.830542 sleeve ->[192.168.10.135:6783|96:ba:36:60:bc:31(h-mul-worker-edge-02)]: Effective MTU verified at 1464
INFO: 2018/05/08 02:59:00.019088 ->[192.168.10.133:34488] connection accepted
INFO: 2018/05/08 02:59:00.022479 ->[192.168.10.133:34488|9a:b2:24:e2:69:b1(h-mul-control-worker-edge-storage-03)]: connection ready; using protocol version 2

My docker config
OPTIONS='--selinux-enabled=false --storage-driver=overlay2 --storage-opt=overlay2.override_kernel_check=true --graph=/data0/docker --live-restore --mtu=1550'

attach file is full log
weave-log.txt

we use ansible to start weave. When modify weave config, we run "systemctl daemon-reload""systemctl enable weave""systemctl restart weave" to wait weave startup. After that we run "weave reset --force" to reset weave and then run "weave expose" to expose containers in the default subnet

[bboreham]

The MTU is set on the weave bridge device - this is why you see it remembered between starts unless you reboot. You can also run weave reset to clear it out before starting.

What you see in the logs are reports from the "sleeve" encapsulation which can negotiate the MTU dynamically along a path. Most of your weave status reports are from the "fastdp" encapsulation which is not dynamic - it either works at a given setting or it doesn't work and falls back to sleeve.

As to why it is different on different VMs I would surmise the settings are different.

[deshui123]

@bboreham
Thanks for your info.
In my ansible task, I try "weave reset" before starting, but it still not work. I still need to reboot.
And for my first deploy(the first time to start weave after i have changed MTU to 1464 in /etc/systemd/system/weave.service), the MTU is always 1376. Until reboot, the MTU can change to 1464.
Can i change MTU successfully except reboot?

main.yml of weave task

• name: generate systemd unit file
  become: yes
  template:
  src: weave.service.j2
  dest: /etc/systemd/system/weave.service
  tags:

  • weave

• name: reload systemd daemon
  become: yes
  shell: systemctl daemon-reload
  tags:

  • weave

• name: enable weave
  become: yes
  shell: "systemctl enable weave"
  tags:

  • weave

• name: reset weave to try delete default MTU
  become: yes
  shell: "/bin/weave reset --force"
  tags:

  • weave

• name: start weave
  become: yes
  shell: "systemctl restart weave"
  tags:

  • weave

• name: wait for weave startup
  become: yes
  tags:

  • weave

• name: reset weave
  become: yes
  shell: "/bin/weave reset --force"
  tags:

  • weave

---

/etc/systemd/system/weave.service

Environment="CHECKPOINT_DISABLE=1"
Environment="WEAVE_MTU=1464"

ExecStartPre=-/usr/bin/docker rm -f weave
ExecStartPre=/usr/bin/weave launch --no-restart --no-dns --ipalloc-range 192.168.1.0/24 --password $WEAVE_PASSWORD $PEERS
ExecStart=/usr/bin/docker attach weave
ExecStop=/usr/bin/weave stop
ExecStopPost=-/usr/bin/docker stop -t 5 weave

[deshui123]

When i run weave expose before "/bin/weave reset --force" in main.yml, the MTU will change to 1464 successfully.

• name: expose containers in the default subnet
  become: yes
  shell: "timeout 20 weave expose"
  retries: 42
  tags:

  • weave

• name: reset weave to try delete default MTU
  become: yes
  shell: "/bin/weave reset --force"
  tags:

  • weave

[bboreham]

I become more puzzled as you add more symptoms. There is no way that weave expose should be able to change the MTU since it is implemented by calling in to the daemon.

Could you fill in some of the basic information requested in the issue template, e.g. Weave Net version, linux version.

To debug this further, can you please recreate the conditions where weave reset does not work, then set the environment variable WEAVE_DEBUG and capture the output of both weave reset and weave launch, then upload it so I can see what is happening.

[deshui123]

@bboreham @brb
Sorry to puzzle you.
If i run weave reset without weave expose, MTU can change successfully. such as:

`name: wait for weave startup
become: yes

name: reset weave
become: yes
shell: "/bin/weave reset --force"`

But if i run weave expose after weave reset, the MTU will not change to 1464,except i run weave reset again or reboot. such as:

`name: wait for weave startup
become: yes

name: reset weave
become: yes
shell: "/bin/weave reset --force"

name: expose containers in the default subnet
become: yes
shell: "timeout 20 weave expose"
retries: 42`

we use weave expose to set ip on the weave bridge. Is there any way to set MTU success without reboot?

$ weave version weave script 2.1.3 weave 2.1.3 $ uname -a Linux h-mul-worker-edge-02 3.10.0-693.17.1.el7.x86_64 #1 SMP Sun Jan 14 10:36:03 EST 2018 x86_64 x86_64 x86_64 GNU/Linux

[bboreham]

$ weave version weave script 2.1.3

That's five months old; there have been three releases since.

If you provide those logs it might be possible to see what is happening.

[deshui123]

Hi @bboreham ,
Thanks for your patient response. I think i have found the root cause.
Since the weave version is 2.1.3, the para "WEAVE_MTU" is used for docker run.
I change the WEAVE_MTU in /usr/bin/weave instead of /etc/systemd/system/weave.service. At this time, MTU is changed successfully without reboot, even if weave starts for the first time.