Two UPFs on same host not working (any more)

[ralfkundel]

Hi,
I'm running two UPFs on the same host listening on the same interface (but on different IPs).

Unfortunately only the UPF started second will receive GTP packets. I assume the second UPF is kicking out the first UPF. I already tried the newest version of upf from the main branch together with the newest kernel module (v0.3.1).

Can someone confirm this bug?

I started them as follows:

sudo -E NFs/upf/build/bin/free5gc-upfd -f config/upfcfg_ralf.yaml &
PID_LIST+=($!)
sleep 1

sudo -E NFs/upf/build/bin/free5gc-upfd -f config/upfcfg_ims_ralf.yaml & 
PID_LIST+=($!)
sleep 1

upfcfg_ims_ralf.yaml.txt
upfcfg_ralf.yaml.txt

[muthuramanecs03g]

Hi @ralfkundel,

Your confguration of two UPF looks good.

Could you please capture the two UPF run log with "trace".

[ralfkundel]

Hi,
attached the two log files with trace. Note that both UPFs are listening on the same interface with diferent IP addresses. I have not tested it with two interfaces on one machine.

upf1.txt
upf2.txt

For me, there is no error detectable. Note that the following script works:
run_dual_upf.sh.txt
the difference is this script creates a namespace and starts the second UPF within the namespace. This makes setting up all the routes more complicated but is a workaround for now ;-)

best and thanks,
Ralf

[muthuramanecs03g]

Hi Ralf ,

Thanks for the update, I will let you know once i verified the given log

Thanks & Regards,
Muhturaman

[muthuramanecs03g]

Hi Ralf,

I verified your work around with UPF source code.

In function _gtp_dev_create(), by default the dest_ns is -1, and it may cause your problem. Because, both interface referenced same namespace which results forwarded the packets to one interface as you mentioned.

Unfortunately only the UPF started second will receive GTP packets. I assume the second UPF is kicking out the first UPF. I >>>already tried the newest version of upf from the main branch together with the newest kernel module (v0.3.1).

static int _gtp_dev_create(int dest_ns, const char *gtp_ifname, int fd, enum ifla_gtp5g_role role)

if (dest_ns >= 0)

mnl_attr_put_u32(nlh, IFLA_NET_NS_FD, dest_ns);

It can be fixed by passing the dest_ns (for the flag IFLA_NET_NS_FD) value in function Gtp5gDeviceAdd(),
status = gtp_dev_create(-1, dev->deviceID, gtp5gDevice.sock->fd);

This fix required to modify the config of UPF YAML as well parse it.

[free5gc-org]

@ralfkundel

If you want to run 2 UPFs on the same machine, you need to create new network namespace.
Or run UPFs on different VMs/docker containers.
This is deployment issue, not UPF issue.
You can try and report us which way is workable.
You can also discuss the deployment issue on free5gc forum

[ralfkundel]

Hi,
Thanks. Yes, i can confirm all 3 approaches to work.

1. Namespaces are the easiest solution, however, the kernel version must be the right one for the gtp kernel module.
2. Using VMs worked on our lab as well. This solution is very nice as you can run any kernel on the hypervisor Independent to the VM. However, configuring this takes some time ;-)
3. Docker we only tested with the same kernel on the hypervisor. But I don‘t see any issues in using different kernel versions. Configuration overhead is similar to a VM but you can script it much better/faster.
   Alternative approach 4: just take two servers. This is the easiest one to debug ;-)

best and thanks