Metallb denied by apparmor on v1.34.1

[jadams]

Summary

Apparmor denies metallb's raw bind request to reply to arps

What Should Happen Instead?

Metallb needs to raw bind to send arps

Reproduction Steps

1. microk8s enable metallb
2. dmesg
3. audit: type=1400 audit(1758143643.160:1674): apparmor="DENIED" operation="bind" class="net" profile="cri-containerd.apparmor.d" pid=39088 comm="speaker" family="packet" sock_type="raw" protocol=0 requested="bind" denied="bind"

Introspection Report

Can you suggest a fix?

fix apparmor rules changed in v1.34.1

Are you interested in contributing with a fix?

no found a fix so I guess yes

[jadams]

offending commit: 3398719

just need to add
network raw,
to microk8s-resources/containerd-profile

[strainovic]

Until the official fix is released, here is a workaround:

sudo vim /etc/apparmor.d/cri-containerd.apparmor.d

Below:

  network inet,
  network inet6,
  network unix,
  network netlink,

Add this line:

network raw,

Reload apparmor:

sudo apparmor_parser -r /etc/apparmor.d/cri-containerd.apparmor.d

Repeat this on each microk8s node. Check the metallb speaker pods, CPU should drop to near 0.

p.s.

Not sure if Snap will revert it to the default, but this works for me (at least for now).

Thanks

[alexray92]

Just ran into this issue as well

[d-shehu]

While it's easy to work around this issue with systemd service, when can we expect this fix to be in the next release?

It does effectively break Metallb at least for me. Thanks.

[taintedkernel]

I hit this yesterday as well, but the workaround resolved the issue for the moment. I'll continue to monitor its status.

[tasnyc]

Woke up to my fans going full blast and and nodes at 100% CPU . Ended up tracking down the errors from the speaker pod logs bind: permission denied then had to do more digging to find out it was an apparmor change that I assume was made in v1.34.1. Things were working fine in 1.34.0.

Made the changes @strainovic mentioned and it seems to have fixed the issue but i really hate having to manually change things so i hope they fix this soon.

Once this gets fixed, which i hope is in the next version I'm thinking of moving away from the plugin and helm deploying the metellb chart as its up to 0.15.2 yet the plugin is 0.13.3 which came out around mid 2022.

If anyone has updates on this issue please post.
Thanks!

[q84fh]

I had same issue as above, adding network raw, to app armor profile helped but would love to see permanent solution!

[alexray92]

Woke up to my fans going full blast and and nodes at 100% CPU

Oh, interesting, I had the same thing! 100% pegged CPU, rebooted, then noticed the issue with metallb.

Funny enough, this particular sort of issue with metallb (problems with ARP advertisement) has been a long-standing problem with microk8s in my experience.

[ngreatorex]

Can confirm I also had this issue. I changed the apparmor profile into complain mode instead of editing it. Probably slightly less secure (and won't persist across reboots) but doesn't involve modifying the snap provided file.

[freddag]

Can confirm that I am also running into this issue, took a while to find that app amour was causing problems, and it doesn't look like @strainovic's workaround persists across reboots, it appears the snap overwrites the config in my single-node dev box on reboot.

[tasnyc]

@freddag ahh i just rebooted my 3 node cluster and it didn't persist so needed to redo @strainovic fix for each node again.
I guess i need to keep this in mind until they address this issue.

[eivindveg]

This sadly is the kind of issue I've spent the last 48 hours obsessing about without getting any closer until I found this issue. Sad to see it hasn't been hotfixed two weeks after proposed solution.

Edit: it also persisted into k3s when checking to see if it was a microk8s issue.

[JPMinAR]

I spent 48 hours trying to locate this issue. Wasn't until I saw logs from:
journalctl -xe | grep -i "denied"
That I shifted from trying to add any number of things to the pod to give it access and finally pivoted to this being something blocked by the AppArmor security.