Network issue when colocating k8s and lxd

[shundezhang]

Bug Description

In a typical COS deployment, k8s and ceph are deployed on the same node: https://pastebin.ubuntu.com/p/XKhqw7bJjZ/
So ceph-mon is deployed in a lxd container so that it is separated from ceph-osd.
Now the problem is, from the lxd container (where ceph-mon runs) it cannot reach traefik LB address on k8s.
K8s is Charmed Canonical k8s with cilium CNI.
When curl the traefik LB address in lxd container, it hangs and eventually times out.

root@juju-13c073-0-lxd-0:~# curl 10.250.120.100/cos-lite-grafana
curl: (28) Failed to connect to 10.250.120.100 port 80 after 129173 ms: Connection timed out

Interestingly, when colocating microk8s and lxd, this works: the lxd container can reach metallb's LB address.
Since microk8s uses Calico, maybe some config in cilium needs adjusted for this to work.

To Reproduce

1. Set up MaaS, and create 3 machines for this (4 cores, 8GB RAM). Each machine should have 2 disks, as one is for ceph osd. They are tagged with "k8s".
2. Deploy k8s and ceph: https://pastebin.ubuntu.com/p/QCrVWTqZ5Y/
3. Add k8s to juju as cloud

juju exec  -u k8s/0 -- sudo k8s config | tee kubeconfig.yaml
KUBECONFIG=kubeconfig.yaml juju add-k8s k8s-cos --client --controller foundations-maas

4. Deploy cos: https://pastebin.ubuntu.com/p/xBhfPrP9Fz/
5. SSH into one ceph-mon unit, and curl traefik LB's address.

Environment

Please see above

Relevant log output

Please see above

Additional context

No response

[berkayoz]

Hey @shundezhang,

Based on the setup you've provided I was able to replicate this issue and look into it.
After testing various scenarios I was able to pinpoint the issue down to the specific setup of lxd container and host using the same juju space.

For this juju sets up a bridge/veth pair, e.g. br-enp5s0. This makes things trickier in relation to Cilium's bpf attachments and features. The main conflict stems from the socketlb feature where the traffic is intercepted/routed incorrectly. The solution seems to be setting socketLB.hostNamespaceOnly=true to pass packets down the stack.

We'll need to evaluate if this is something we want to set by default or enable through an extra annotation. I'll update here when we arrive at a solution, thanks!