netoffload

netoffload is a simple toolkit for simplifying the process of managing network offloading features on Linux. It aims at abstracting all of the complexity involved in managing these features, and providing a simple interface for enabling and disabling them.

Usage

Before you can use offloadctl, you must first ensure that a few dependencies are setup on your system. However, don't worry, offloadctl will tell you exactly what you need to do if it notices that something is missing.

BIOS configuration

Depending on your hardware, you may need to enable a few options in the BIOS in order to get SR-IOV working. You should consult with your hardware vendor to ensure that you have all the options but you can use the following as a guideline.

Supermicro

You need to make sure that you have the latest BIOS installed on your system and that you have the following options enabled:

• Advanced > ACPI settings > PCI AER Support > Enabled (if you're using a H12/H11 series motherboard with Rome CPU such as EPYC 7xx2)
• Advanced > CPU configuration > SVM Mode > Enabled
• Advanced > NB Configuration > IOMMU > Enabled
• Advanced > NB Configuration > ACS Enable > Enabled
• Advanced > PCIe/PCI/PnP Configuration > SR-IOV Support > Enabled

Kernel configuration

You must have the following kernel options enabled in your command line:

• iommu=pt
• intel_iommu=on (for systems with Intel CPUs)
• amd_iommu=on (for systems with AMD CPUs)

The most common way to do this is to edit /etc/default/grub and add them to the GRUB_CMDLINE_LINUX_DEFAULT variable, then run update-grub to update your grub configuration.

NIC configuration

There are a few steps necessary to configure your NIC for hardware acceleration. The exact steps will vary depending on your NIC vendor and model, but the following can be used as a guideline.

Mellanox

1. Install the mstflint tools on the compute node which will be used:

   sudo apt-get install mstflint

2. Get the device's PCI by using lspci.

   $ lspci | grep Mellanox
   61:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]
   61:00.1 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx]

3. Check if SR-IOV is enabled in the firmware

   $ sudo mstconfig -d 61:00.0 q | grep SRIOV_EN
        SRIOV_EN                            True(1)

   If SR-IOV is not enabled, you can enable it with the following command:

   $ sudo mstconfig -d 61:00.0 set SRIOV_EN=1

4. Configure the needed number of VFs

   $ sudo mstconfig -d 61:00.0 set NUM_OF_VFS=16

5. Restart the system

   Note

   A useful tip is to prefix this command with an extra space (before sudo), so that it is not saved in the shell history and prevents accidental reboot.

   $ sudo reboot
   

Enabling ASAP2

ASAP2 is a Mellanox-specific network hardware acceleration feature. It can be enabled by running the following command:

$ sudo offloadctl asap2 enable enp97s0f0 --vfs 16

If you're running containerd on the host, you can also use ctr to run the container:

$ ctr run \
    --rm \
    --net-host \
    --privileged \
    --mount type=bind,src=/run,dst=/run,options=rbind:rw \
    ghcr.io/vexxhost/netoffload:latest \
    netoffload \
    offloadctl enable asap2 enp97s0f0 --vfs 16

This will enable ASAP2 on the given device (enp97s0f0 in this example) and create 16 VFs. Once completed, it ensures that the other-config:hw-offload is set to true in the Open vSwitch database.

If it detects that the device is not configured for SR-IOV, it will automatically enable it for you or prompt you to do so for steps involving reboots.