Skip to content

Commit 27a6246

Browse files
RamyaGuruclaude
andcommitted
Update system config tutorial for DAQIRI migration
Rebrand tutorial from Holoscan Advanced Networking to DAQIRI throughout the intro, prerequisites, and background sections. Add a "Plan your reboots" tip before Section 3 tuning steps and a Section 3.10 summary table mapping each tuning step to its tune_system.py flag and whether a persistent option is available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ramya Gurunathan <rgurunathan@nvidia.com>
1 parent 42ecfde commit 27a6246

1 file changed

Lines changed: 30 additions & 8 deletions

File tree

docs/tutorials/system_config/README.md

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
1-
# High Performance Networking with Holoscan
1+
# System Setup for DAQIRI
22

3-
This tutorial demonstrates how to use the Advanced Network library (referred to as `advanced_network` in HoloHub) for low latency and high throughput communication through NVIDIA SmartNICs. With a properly tuned system, the Advanced Network library can achieve hundreds of Gbps with latencies in the low microseconds.
3+
This tutorial demonstrates how to use the DAQIRI library for low latency and high throughput communication through NVIDIA SmartNICs. With a properly tuned system, DAQIRI can achieve hundreds of Gbps with latencies in the low microseconds.
44

55
!!! note
66

7-
This solution is designed for users who want to create a Holoscan application that will interface with an external system or sensor over Ethernet.
7+
This solution is designed for users who want to create an application that interfaces with an external system or sensor over Ethernet.
88

99
- For high performance communication with systems also running Holoscan, refer to the [Holoscan distributed application documentation](https://docs.nvidia.com/holoscan/sdk-user-guide/holoscan_create_distributed_app.html) instead.
10-
- For JESD-compliant sensor without Ethernet support, consider the [Holoscan Sensor Bridge](https://docs.nvidia.com/holoscan/sensor-bridge/latest/introduction.html) for an FPGA-based interface to Holoscan.
10+
- For JESD-compliant sensors without Ethernet support, consider the [Holoscan Sensor Bridge](https://docs.nvidia.com/holoscan/sensor-bridge/latest/introduction.html) for an FPGA-based interface.
1111

1212
## Prerequisites
1313

14-
Achieving High Performance Networking with Holoscan requires a system with an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) and a [**discrete GPU**](https://www.nvidia.com/en-us/design-visualization/desktop-graphics/). That is the case of [NVIDIA Data Center](https://www.nvidia.com/en-us/data-center/) systems, or edge systems like the [NVIDIA IGX](https://www.nvidia.com/en-us/edge-computing/products/igx/) platform and the [NVIDIA Project DIGITS](https://www.nvidia.com/en-us/project-digits/). `x86_64` systems equipped with these components are also supported, though the performance will vary greatly depending on the PCIe topology of the system (more on this [below](#31-ensure-ideal-pcie-topology)).
14+
Achieving high performance networking with DAQIRI requires a system with an [**NVIDIA SmartNIC**](https://www.nvidia.com/en-us/networking/ethernet-adapters/) and a [**discrete GPU**](https://www.nvidia.com/en-us/design-visualization/desktop-graphics/). That is the case of [NVIDIA Data Center](https://www.nvidia.com/en-us/data-center/) systems, or edge systems like the [NVIDIA IGX](https://www.nvidia.com/en-us/edge-computing/products/igx/) platform and the [NVIDIA Project DIGITS](https://www.nvidia.com/en-us/project-digits/). `x86_64` systems equipped with these components are also supported, though the performance will vary greatly depending on the PCIe topology of the system (more on this [below](#31-ensure-ideal-pcie-topology)).
1515

1616
In this tutorial, we will be developing on an **NVIDIA IGX Orin platform** with [IGX SW 1.1](https://docs.nvidia.com/igx-orin/user-guide/latest/base-os.html) and an [NVIDIA RTX 6000 ADA GPU](https://www.nvidia.com/en-us/design-visualization/rtx-6000/), which is the configuration that is currently actively tested. The concepts should be applicable to other systems based on Ubuntu 22.04 as well. It should also work on other Linux distributions with a glibc version of 2.35 or higher by containerizing the dependencies and applications on top of an Ubuntu 22.04 image, but this is not actively tested at this time.
1717

@@ -25,7 +25,7 @@ Achieving high performance networking is a complex problem that involves many sy
2525

2626
### Kernel Bypass
2727

28-
In this context, Kernel Bypass refers to bypassing the operating system's kernel to directly communicate with the network interface (NIC), greatly reducing the latency and overhead of the Linux network stack. There are multiple technologies that achieve this in different fashions. They're all Ethernet-based, but differ in their implementation and features. The goal of the Advanced Network library in Holoscan Networking is to provide a common higher-level interface to all these backends:
28+
In this context, Kernel Bypass refers to bypassing the operating system's kernel to directly communicate with the network interface (NIC), greatly reducing the latency and overhead of the Linux network stack. There are multiple technologies that achieve this in different fashions. They're all Ethernet-based, but differ in their implementation and features. The goal of the DAQIRI library is to provide a common higher-level interface to all these backends:
2929

3030
- **RDMA**: Remote Direct Memory Access, using the open-source [`rdma-core`](https://github.com/linux-rdma/rdma-core) library. It differs from the other Ethernet-based backends with its server/client model and RoCE (RDMA over Ethernet) protocol. Given the extra cost and complexity to setup on both ends, it offers a simpler user interface, orders packets on arrival, and is the only one to offer a high reliability mode.
3131
- **DPDK**: the Data Plane Development Kit is an open-source project part of the Linux Foundation with a strong and long-lasting community support. Its RTE Flow capability is generally considered the most flexible solution to split packets ingress and egress data.
@@ -34,14 +34,14 @@ In this context, Kernel Bypass refers to bypassing the operating system's kernel
3434

3535
??? example "Work In Progress"
3636

37-
The Holoscan Advanced Network library integration testing infrastructure is under active development. As such:
37+
The DAQIRI library integration testing infrastructure is under active development. As such:
3838

3939
- The **DPDK** backend is supported and distributed with the DAQIRI library, and is the only backend actively tested at this time.
4040
- The **DOCA GPUNetIO** backend is supported and distributed with the DAQIRI library, with testing infrastructure under development.
4141
- The **NVIDIA Rivermax** backend is supported for Rx only when building from source, but not yet distributed nor actively tested. Tx support is under development.
4242
- The **RDMA** backend is under active development and should be available soon.
4343

44-
Which backend is best for your use case will depend on multiple factors, such as packet size, batch size, data type, and more. The goal of the Advanced Network library is to abstract the interface to these backends, allowing developers to focus on the application logic and experiment with different configurations to identify the best technology for their use case.
44+
Which backend is best for your use case will depend on multiple factors, such as packet size, batch size, data type, and more. The goal of the DAQIRI library is to abstract the interface to these backends, allowing developers to focus on the application logic and experiment with different configurations to identify the best technology for their use case.
4545

4646
### GPUDirect
4747

@@ -475,6 +475,10 @@ If it's not loaded, run the following command, then check again:
475475

476476
While the configurations above are the minimum requirements to get a NIC and a NVIDIA GPU to communicate while bypassing the OS kernel stack, performance can be further improved in most scenarios by tuning the system as described below.
477477

478+
!!! tip "Plan your reboots"
479+
480+
Several steps below require adding flags to the kernel bootline in `/etc/default/grub` (hugepages in [3.4](#34-enable-huge-pages), CPU isolation in [3.5](#35-isolate-cpu-cores)). We recommend reading through both sections first and adding all the flags at once to avoid multiple reboots. Other items like MRRS, GPU clocks, and MTU can be applied at runtime but reset on reboot — consider scripting them or using a systemd service for persistence.
481+
478482
Before diving in each of the setups below, we provide a utility script as part of the DAQIRI library which provides an overview of the configurations that potentially need to be tuned on your system.
479483

480484
??? example "Work In Progress"
@@ -1479,6 +1483,24 @@ You can set the MTU for each interface like so, for a given `if_name` name ident
14791483
maxmtu 9978
14801484
```
14811485
1486+
### 3.10 Summary
1487+
1488+
| Step | Description | Tuning Script Flag | Persistent Option Available? |
1489+
|------|-------------|--------------------|-------------|
1490+
| 3.1 | PCIe topology | `--check topo` | N/A (hardware) |
1491+
| 3.2 | PCIe config (MPS/Speed) | `--check mps` | N/A (hardware) |
1492+
| 3.3 | NIC MRRS | `--check mrrs` / `--set mrrs` | No — use a startup script |
1493+
| 3.4 | Hugepages | `--check hugepages` | Yes — kernel bootline or `/etc/fstab` |
1494+
| 3.5 | CPU isolation | `--check cmdline` | Yes — kernel bootline |
1495+
| 3.6 | CPU governor | `--check cpu-freq` | Yes — see persistent option in section |
1496+
| 3.7 | GPU clocks | `--check gpu-clock` | Partial — `nvidia-smi -pm 1` persists driver; clock locks need a startup script |
1497+
| 3.8 | GPU BAR1 size | `--check bar1-size` | Yes — firmware flash |
1498+
| 3.9 | Jumbo frames (MTU) | `--check mtu` | Yes — see persistent option in section |
1499+
1500+
!!! tip
1501+
1502+
Each section above provides both one-time and persistent configuration options. We recommend testing with the one-time commands first, then switching to the persistent options once your configuration is verified. You can check all settings at once with `tune_system.py --check all`.
1503+
14821504
## 4. Running a test application
14831505
14841506
Holoscan Networking provides a benchmarking application named `adv_networking_bench` that can be used to test the performance of the networking configuration. In this section, we'll walk you through the steps needed to configure the application for your NIC for Tx and Rx, and run a loopback test between the two interfaces with a [physical SFP cable](https://www.nvidia.com/en-us/networking/interconnect/) connecting them.

0 commit comments

Comments
 (0)