ipset: v6.29: Set cannot be destroyed: it is in use by a kernel component

[david-rooke]

Environment:

• kubernetes: 1.6.0
• Alpine Linux: 3.4.6
• Weave: 1.9.5

I am seeing regular weave-npc crashes relating to ipset:

INFO: 2017/05/09 07:58:21.961956 destroying ipset: &npc.selectorSpec{key:"", selector:labels.internalSelector{}, ipsetType:"hash:ip", ipsetName:"weave-94D4P%N1ac^8#m}i;KwFqXYQu"}
FATA: 2017/05/09 07:58:21.977964 ipset [destroy weave-94D4P%N1ac^8#m}i;KwFqXYQu] failed: ipset v6.29: Set cannot be destroyed: it is in use by a kernel component
: exit status 1

[stuart-warren]

related to #2915 ?

[marccarre]

It would appear as a duplicate, yes.

Closing for now, feel free to comment on #2915 directly, or to re-open this issue if you think they are distinct.

@david-rooke,

1. did you try the workaround from #2915?
2. you may also want to look for IPTables rules (iptables-save) which are using such an ipset, and delete these.

[stuart-warren]

I actually now think this is a separate issue, can it be reopened?

#2915 occurs on startup and is resolved by #2949, running Flush before Destroy

#2939 (this issue) occurs during runtime and is likely due to this Destroy method: https://github.com/weaveworks/weave/blob/1.9/npc/selector.go#L100 failing.

[mikebryant]

Thought - this is probably because we have an aggressive runtime end-to-end test going, which among other things, is creating/deleting namespaces a lot, so I think we're exercising a code path that's probably not that well tested, or used as frequently elsewhere

[mikebryant]

We're also seeing things like:

FATA: 2017/05/12 15:38:51.072861 add namespace: exit status 4: iptables: Resource temporarily unavailable.

Both of these issues appear to be somewhat racy, the same namespace can be brought up or taken down on some nodes and not others

[bboreham]

We have code that retries an iptables operation until that doesn't happen; it will be easy to call that from weave-npc and/or move the code up to the coreos iptables lib.

I did have a brief discussion with someone from CoreOS who told me that could never happen, so it's interesting to see you hit it.

[mikebryant]

On resource temporarily unavailable:
I think this issue is related: kubernetes/kubernetes#43575

In my coreos environment the iptables versions are:
coreos 1235.9.0: 1.4.21
kube-proxy container: 1.4.21
weave container: 1.6.0

The issues I can see:

• The version of iptables in weave uses a different locking mechanism (/run/xtables.lock) to the one running on the host, or in kube-proxy
• Weave isn't mounting in /run/xtables.lock in any case, so the locking its iptables invocation is doing doesn't affect the host

On set cannot be destroyed:
I've been looking over the codepaths involved, and I can't see anything wrong. I set up my weave-npc container to just sit waiting after it crashed, and when I exec'ed in and had a poke around, the ipset it failed to destroy had no references, and I was able to just ipset destroy it myself, with no issues.

I'm going to experiment with adding a big sleep in, and see if there's some sort of race condition between removing the other ipset/iptables rules, and the kernel being ready for the namespace ipset to be deleted

[bboreham]

Weave isn't mounting in /run/xtables.lock

that is a valuable observation

[bboreham]

Unfortunately we cannot mount /run (symlinked from /var/run) because that causes other trouble as seen in #1455

So maybe repeating until it doesn't fail is the best way?