You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Update system config tutorial for DAQIRI migration
Rebrand tutorial from Holoscan Advanced Networking to DAQIRI throughout
the intro, prerequisites, and background sections. Add a "Plan your
reboots" tip before Section 3 tuning steps and a Section 3.10 summary
table mapping each tuning step to its tune_system.py flag and whether
a persistent option is available.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
Copy file name to clipboardExpand all lines: docs/tutorials/system_config/README.md
+30-8Lines changed: 30 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,17 +1,17 @@
1
-
# High Performance Networking with Holoscan
1
+
# System Setup for DAQIRI
2
2
3
-
This tutorial demonstrates how to use the Advanced Network library (referred to as `advanced_network` in HoloHub) for low latency and high throughput communication through NVIDIA SmartNICs. With a properly tuned system, the Advanced Network library can achieve hundreds of Gbps with latencies in the low microseconds.
3
+
This tutorial demonstrates how to use the DAQIRI library for low latency and high throughput communication through NVIDIA SmartNICs. With a properly tuned system, DAQIRI can achieve hundreds of Gbps with latencies in the low microseconds.
4
4
5
5
!!! note
6
6
7
-
This solution is designed for users who want to create a Holoscan application that will interface with an external system or sensor over Ethernet.
7
+
This solution is designed for users who want to create an application that interfaces with an external system or sensor over Ethernet.
8
8
9
9
- For high performance communication with systems also running Holoscan, refer to the [Holoscan distributed application documentation](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_distributed_app.html) instead.
10
-
- For JESD-compliant sensor without Ethernet support, consider the [Holoscan Sensor Bridge](https://docs.nvidia.com/holoscan/sensor-bridge/latest/introduction.html) for an FPGA-based interface to Holoscan.
10
+
- For JESD-compliant sensors without Ethernet support, consider the [Holoscan Sensor Bridge](https://docs.nvidia.com/holoscan/sensor-bridge/latest/introduction.html) for an FPGA-based interface.
11
11
12
12
## Prerequisites
13
13
14
-
Achieving High Performance Networking with Holoscan requires a system with an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) and a [**discrete GPU**](https://www.nvidia.com/en-us/design-visualization/desktop-graphics/). That is the case of [NVIDIA Data Center](https://www.nvidia.com/en-us/data-center/) systems, or edge systems like the [NVIDIA IGX](https://www.nvidia.com/en-us/edge-computing/products/igx/) platform and the [NVIDIA Project DIGITS](https://www.nvidia.com/en-us/project-digits/). `x86_64` systems equipped with these components are also supported, though the performance will vary greatly depending on the PCIe topology of the system (more on this [below](#31-ensure-ideal-pcie-topology)).
14
+
Achieving high performance networking with DAQIRI requires a system with an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) and a [**discrete GPU**](https://www.nvidia.com/en-us/design-visualization/desktop-graphics/). That is the case of [NVIDIA Data Center](https://www.nvidia.com/en-us/data-center/) systems, or edge systems like the [NVIDIA IGX](https://www.nvidia.com/en-us/edge-computing/products/igx/) platform and the [NVIDIA Project DIGITS](https://www.nvidia.com/en-us/project-digits/). `x86_64` systems equipped with these components are also supported, though the performance will vary greatly depending on the PCIe topology of the system (more on this [below](#31-ensure-ideal-pcie-topology)).
15
15
16
16
In this tutorial, we will be developing on an **NVIDIA IGX Orin platform** with [IGX SW 1.1](https://docs.nvidia.com/igx-orin/user-guide/latest/base-os.html) and an [NVIDIA RTX 6000 ADA GPU](https://www.nvidia.com/en-us/design-visualization/rtx-6000/), which is the configuration that is currently actively tested. The concepts should be applicable to other systems based on Ubuntu 22.04 as well. It should also work on other Linux distributions with a glibc version of 2.35 or higher by containerizing the dependencies and applications on top of an Ubuntu 22.04 image, but this is not actively tested at this time.
17
17
@@ -25,7 +25,7 @@ Achieving high performance networking is a complex problem that involves many sy
25
25
26
26
### Kernel Bypass
27
27
28
-
In this context, Kernel Bypass refers to bypassing the operating system's kernel to directly communicate with the network interface (NIC), greatly reducing the latency and overhead of the Linux network stack. There are multiple technologies that achieve this in different fashions. They're all Ethernet-based, but differ in their implementation and features. The goal of the Advanced Network library in Holoscan Networking is to provide a common higher-level interface to all these backends:
28
+
In this context, Kernel Bypass refers to bypassing the operating system's kernel to directly communicate with the network interface (NIC), greatly reducing the latency and overhead of the Linux network stack. There are multiple technologies that achieve this in different fashions. They're all Ethernet-based, but differ in their implementation and features. The goal of the DAQIRI library is to provide a common higher-level interface to all these backends:
29
29
30
30
-**RDMA**: Remote Direct Memory Access, using the open-source [`rdma-core`](https://github.com/linux-rdma/rdma-core) library. It differs from the other Ethernet-based backends with its server/client model and RoCE (RDMA over Ethernet) protocol. Given the extra cost and complexity to setup on both ends, it offers a simpler user interface, orders packets on arrival, and is the only one to offer a high reliability mode.
31
31
-**DPDK**: the Data Plane Development Kit is an open-source project part of the Linux Foundation with a strong and long-lasting community support. Its RTE Flow capability is generally considered the most flexible solution to split packets ingress and egress data.
@@ -34,14 +34,14 @@ In this context, Kernel Bypass refers to bypassing the operating system's kernel
34
34
35
35
??? example "Work In Progress"
36
36
37
-
The Holoscan Advanced Network library integration testing infrastructure is under active development. As such:
37
+
The DAQIRI library integration testing infrastructure is under active development. As such:
38
38
39
39
- The **DPDK** backend is supported and distributed with the DAQIRI library, and is the only backend actively tested at this time.
40
40
- The **DOCA GPUNetIO** backend is supported and distributed with the DAQIRI library, with testing infrastructure under development.
41
41
- The **NVIDIA Rivermax** backend is supported for Rx only when building from source, but not yet distributed nor actively tested. Tx support is under development.
42
42
- The **RDMA** backend is under active development and should be available soon.
43
43
44
-
Which backend is best for your use case will depend on multiple factors, such as packet size, batch size, data type, and more. The goal of the Advanced Network library is to abstract the interface to these backends, allowing developers to focus on the application logic and experiment with different configurations to identify the best technology for their use case.
44
+
Which backend is best for your use case will depend on multiple factors, such as packet size, batch size, data type, and more. The goal of the DAQIRI library is to abstract the interface to these backends, allowing developers to focus on the application logic and experiment with different configurations to identify the best technology for their use case.
45
45
46
46
### GPUDirect
47
47
@@ -475,6 +475,10 @@ If it's not loaded, run the following command, then check again:
475
475
476
476
While the configurations above are the minimum requirements to get a NIC and a NVIDIA GPU to communicate while bypassing the OS kernel stack, performance can be further improved in most scenarios by tuning the system as described below.
477
477
478
+
!!! tip "Plan your reboots"
479
+
480
+
Several steps below require adding flags to the kernel bootline in`/etc/default/grub` (hugepages in [3.4](#34-enable-huge-pages), CPU isolation in [3.5](#35-isolate-cpu-cores)). We recommend reading through both sections first and adding all the flags at once to avoid multiple reboots. Other items like MRRS, GPU clocks, and MTU can be applied at runtime but reset on reboot — consider scripting them or using a systemd service for persistence.
481
+
478
482
Before diving in each of the setups below, we provide a utility script as part of the DAQIRI library which provides an overview of the configurations that potentially need to be tuned on your system.
479
483
480
484
??? example "Work In Progress"
@@ -1479,6 +1483,24 @@ You can set the MTU for each interface like so, for a given `if_name` name ident
| 3.9 | Jumbo frames (MTU) |`--check mtu`| Yes — see persistent option in section |
1499
+
1500
+
!!! tip
1501
+
1502
+
Each section above provides both one-time and persistent configuration options. We recommend testing with the one-time commands first, then switching to the persistent options once your configuration is verified. You can check all settings at once with `tune_system.py --check all`.
1503
+
1482
1504
## 4. Running a test application
1483
1505
1484
1506
Holoscan Networking provides a benchmarking application named `adv_networking_bench` that can be used to test the performance of the networking configuration. In this section, we'll walk you through the steps needed to configure the application for your NIC for Tx and Rx, and run a loopback test between the two interfaces with a [physical SFP cable](https://www.nvidia.com/en-us/networking/interconnect/) connecting them.
0 commit comments