SCION-IP-Gateway (SIG) not working on Megalos (Kubernetes) despite working on Kathará (Docker)

[MOG-22]

Dear Kathara Team,

I'm trying to set up a Kathara Lab that emulates a small SCION topology, where each SCION AS has an IP endhost that communicates with each other over SCION using the SCION IP gateway.

With the recent changes to the standard SCION image, by changing the permission of the container and adding the tunnel device to the container:

# Adjust scion-ip-gateway.service for compatibility with docker environment
RUN sed -i 's/^User=scion/User=root/' /usr/lib/systemd/system/scion-ip-gateway.service && \
   sed -i 's/^Group=scion/Group=root/' /usr/lib/systemd/system/scion-ip-gateway.service && \
   sed -i '/^AmbientCapabilities=cap_net_admin/d' /usr/lib/systemd/system/scion-ip-gateway.service && \
   sed -i '/^\[Service\]/a ExecStartPre=\/bin\/bash -c '"'"'mkdir -p /dev/net; mknod /dev/net/tun c 10 200; chmod 600 /dev/net/tun'"'"'' /usr/lib/systemd/system/scion-ip-gateway.service

The SIG, and therefore the Lab, works fine on Docker. But as soon as you try to run the lab in Megalos (Kubernetes), the communication between the IP end hosts (e.g. pinging) does not work anymore.

I'm new to Kubernetes and hope you can help me find out why Megalos currently doesn't support this and how it could be fixed. At the moment, I think this issue is related to the fact that we need to change the way we set up the tunnel on Kubernetes due to the different networking in Kubernetes. Do you have ideas what we need to change ?

Thank you in advance for your reply!

[Skazza94]

Hi @MOG-22,
can you please share the lab that you are trying to deploy?
It would be easier for us to identify the problem.

Cheers,
Mariano

[MOG-22]

Hi @Skazza94,
thanks for your fast reply. I already found the mistake. It had nothing to do with the tunnel. Rather it seemed that the default route I configured in the IP-Endhosts .startup file (such that they send all their traffic to the respective SCION AS they belong to) got "removed" (wasn't showing up anymore when running "ip route" in the device) when running it on Megalos.

Executing the "ip route add default ..." command on the IP-Endhosts resolved the issue, and they were again able to communicate with each other.

Do you have you an idea where this could come from, that the default route gets "lost"?

Best,
Marc-Oliver

[Skazza94]

Hi @MOG-22,
while using Megalos you should remap all your interfaces from the prefix eth to net. So, for example, if your interface is eth0, you should rename it in net0 while using Megalos. This is documented in the dedicated Wiki page: https://github.com/KatharaFramework/Kathara/wiki/Megalos-(Kathara-over-Kubernetes).

In this specific case, we remove all the default routes registered to eth0 (since they are used by the internal K8s network).

Let me know if the problem persists.

Cheers,
Mariano.