weave failed to start - systemd - CentOS 7 - SELinux involved

[karaktaka]

I'm trying to start weave with systemd and it keeps failing.

`● weave.service - Weave Network
Loaded: loaded (/etc/systemd/system/weave.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Thu 2017-03-23 11:29:19 CET; 1s ago
Docs: http://docs.weave.works/weave/latest_release/
Process: 24554 ExecStartPre=/usr/local/bin/weave launch-router $WEAVE_ROUTER_OPTS (code=exited, status=203/EXEC)
Main PID: 22981 (code=exited, status=1/FAILURE)

Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: Failed to start Weave Network.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: Unit weave.service entered failed state.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: weave.service failed.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: weave.service holdoff time over, scheduling restart.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: start request repeated too quickly for weave.service
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: Failed to start Weave Network.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: Unit weave.service entered failed state.
Mar 23 11:29:19 agent01.test.dumontnet.de systemd[1]: weave.service failed.
`

/etc/systemd/system/weave.service
`[Unit]
Description=Weave Network
Documentation=http://docs.weave.works/weave/latest_release/
Requires=docker.service
After=docker.service

[Service]
EnvironmentFile=-/etc/sysconfig/weave
Restart=on-failure
TimeoutStartSec=10
ExecStartPre=/usr/local/bin/weave launch-router $WEAVE_ROUTER_OPTS
ExecStart=/usr/bin/docker $WEAVE_SOCKET attach weave
ExecStop=/usr/local/bin/weave stop

[Install]
WantedBy=weave-network.target`

/etc/systemd/system/weaveproxy.service
`[Unit]
Description=Weave proxy for Docker API
Documentation=http://docs.weave.works/
Requires=docker.service
After=docker.service

[Service]
EnvironmentFile=-/etc/sysconfig/weave
Restart=on-failure
ExecStartPre=/usr/local/bin/weave launch-proxy $WEAVE_SOCKET $WEAVE_PROXY_OPTS
ExecStart=/usr/bin/docker $WEAVE_SOCKET attach weaveproxy
Restart=on-failure
ExecStop=/usr/local/bin/weave stop-proxy

[Install]
WantedBy=weave-network.target`

/etc/systemd/system/weave-network.target
`[Unit]
Description=Weave Network Setup Complete
Documentation=man:systemd.special(7)
RefuseManualStart=no
After=network-online.target
After=weave.service
After=docker.service

[Install]
WantedBy=multi-user.target`

/etc/sysconfig/weave
WEAVE_SOCKET="-H unix:///var/run/weave.sock" WEAVE_PROXY_OPTS="--no-restart" WEAVE_ROUTER_OPTS="--dns-domain=service.test.dumontnet.de. --no-restart --ipalloc-init observer --ipalloc-range 10.10.10.0/24 192.168.104.175 192.168.104.176 192.168.104.177"

Any help is appreciated.

Thanks

[marccarre]

Hi @karaktaka, could you please:

1. share the output of docker logs for the Weave containers? -- these should be more insightful
2. share the output of docker version?
3. share the output of cat /etc/*release*?
4. share the output of uname -a?

[karaktaka]

There are no logs for any containers as they are not created at all.
Seems like weave won't get started at all.

`Client:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64

Server:
Version: 1.11.2
API version: 1.23
Go version: go1.5.4
Git commit: b9f10c9
Built: Wed Jun 1 21:23:11 2016
OS/Arch: linux/amd64`

`CentOS Linux release 7.3.1611 (Core)
Derived from Red Hat Enterprise Linux 7.3 (Source)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.3.1611 (Core)
CentOS Linux release 7.3.1611 (Core)
cpe:/o:centos:centos:7
`

Linux agent01.test.dumontnet.de 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[marccarre]

What happens if you run this manually?

$ WEAVE_ROUTER_OPTS="--dns-domain=service.test.dumontnet.de. --no-restart --ipalloc-init observer --ipalloc-range 10.10.10.0/24 192.168.104.175 192.168.104.176 192.168.104.177"
$ /usr/local/bin/weave launch-router $WEAVE_ROUTER_OPTS 

[karaktaka]

It works fine. Weave starts as usual.

To provide some additional information:

I used this link and the examples there as templates for my systemd units -> https://gist.github.com/errordeveloper/27ef79e5b634065dc91f

and also applied the selinux recommendations from here -> https://www.weave.works/documentation/net-latest-installing-weave/net-latest-systemd/

[marccarre]

@karaktaka, would you mind sharing the full journalctl logs for weave.service and docker.service?
e.g. using:

• $ journalctl -u docker --no-pager
• $ journalctl -u weave --no-pager

[karaktaka]

Sure, but i played with some settings, so there might be some errors which are not relevant. Especially at the beginning.

weave_log.txt
docker_log.txt

[karaktaka]

I also used the guide from here -> https://github.com/weaveworks/guides/tree/master/mesos-marathon/centos and set up a vagrant machine. Works fine there. Tried to apply those configurations to my setup, but it still remains not working.

[karaktaka]

I found the problem. Setting selinux to permissive fixed the problem. Centos in vagrant is also configured with permissive selinux. But the howto at https://www.weave.works/documentation/net-latest-installing-weave/net-latest-systemd/ says, that you can allow weave with selinux, but it doesn't seem to work properly. I don't know selinux enough to correct this so someone may look into this.

[marccarre]

@karaktaka, glad you found a solution!
Would you mind sharing the exact command which fixed it on your end? setenforce 0?

@errordeveloper, would you know more about these instructions?
They seem to come from #325, but there isn't much context on #310, which it fixes.
And given these are more than 2 years old, I am no longer sure they are up-to-date.

Also, though the OP doesn't use Kubernetes, I notice that kubeadm's instructions mention:

Note: Disabling SELinux by running setenforce 0 is required in order to allow containers to access the host filesystem, which is required by pod networks for example.

Weave Net definitely needs to access to /etc/resolv.conf and other files on the host, though I am not sure how Docker handles this and how it integrates with SELinux, and assuming Kubernetes simply rely on Docker to do the right thing, if the above instruction may apply to Weave in general. Thoughts?

[karaktaka]

I added "enforcing=0" to my "GRUB_CMDLINE_LINUX" in /etc/default/grub so selinux will stay in permissive mode even after a reboot.

[marccarre]

Note that while this resolves the issue, it is probably quite unsafe from a SELinux perspective.
@errordeveloper, after having read a bit more about SELinux, unconfined_exec_t has a slightly smaller scope but does not seem much better, we should try to find a better way to do it so that it both works and is secure.

[karaktaka]

@marccarre I agree with you. I would rather use the cluster with selinux enabled than not, but currently it's the only way to get it working. If I find a solution I'll let you know.