Skip to content

Commit 9581cbb

Browse files
authored
fix(nvidia-tuned): remove isolcpus from all inference profiles (#96)
## Summary - Removes `cmdline_isolcpus` from all inference profiles (os/common h100 + gb200, service/aks h100, service/eks h100 + gb200). - Cleans up stale "Without cpu isolation due to tuned package version" summary text in debian/11 and ubuntu/22.04 os-specific gb200 profiles. - Bumps `package_version` to `0.3.2`. ## Background Closes #95. `isolcpus=0,1,48,49` on AKS H100 nodes was isolating the housekeeping cores (including CPU0, the default IRQ target) rather than workload cores. Under high-concurrency inference serving this starved network/softirq processing: | Configuration | Throughput | TTFT p99 | Verdict | |---|---|---|---| | Full profile (with `isolcpus`) | 50,404 tok/s | 6,034 ms | FAIL | | Same profile, `isolcpus` removed | 148,344 tok/s | 614 ms | PASS | | Untuned stock AKS node | 146,452 tok/s | 547 ms | PASS | Single-variable bisect from #95 confirmed `isolcpus` was the sole cause. The remaining profile settings (hugepages, iommu=pt, pci=realloc=off, init_on_alloc=0, vm.swappiness=1) are innocent. ## Test plan - [ ] CI validate + dev build passes for `nvidia-tuned` - [ ] `make test-package PACKAGE=nvidia-tuned` passes locally - [ ] Verify no `isolcpus` in any tuned.conf: `grep -r isolcpus nvidia-tuned/` Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>
1 parent faf01e4 commit 9581cbb

8 files changed

Lines changed: 3 additions & 13 deletions

File tree

nvidia-tuned/config.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"schema_version": "v1",
33
"package_name": "nvidia_tuned",
4-
"package_version": "0.3.1",
4+
"package_version": "0.3.2",
55
"expected_config_files": ["accelerator"],
66
"modes": {
77
"uninstall": [

nvidia-tuned/profiles/os/common/nvidia-gb200-inference/tuned.conf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@ include=nvidia-gb200-performance
33
summary=Optimized for inference workloads
44

55
[bootloader]
6-
# Isolate CPUs for inference processes, 2 per socket
7-
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}
86
# Allocate hugepages for better memory access
97
cmdline_hugepages=hugepagesz=2M hugepages=8192
108

nvidia-tuned/profiles/os/common/nvidia-h100-inference/tuned.conf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@ include=nvidia-h100-performance
33
summary=Optimized for inference workloads
44

55
[bootloader]
6-
# Isolate CPUs for inference processes, 2 per socket
7-
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}
86
# Allocate hugepages for better memory access
97
cmdline_hugepages=hugepagesz=2M hugepages=8192
108

nvidia-tuned/profiles/os/debian/11/nvidia-gb200-inference/tuned.conf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[main]
22
include=nvidia-gb200-performance
3-
summary=Optimized for inference workloads. Without cpu isolation due to tuned package version.
3+
summary=Optimized for inference workloads
44

55
[bootloader]
66
# Allocate hugepages for better memory access

nvidia-tuned/profiles/os/ubuntu/22.04/nvidia-gb200-inference/tuned.conf

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[main]
22
include=nvidia-gb200-performance
3-
summary=Optimized for inference workloads. Without cpu isolation due to tuned package version.
3+
summary=Optimized for inference workloads
44

55
[bootloader]
66
# Allocate hugepages for better memory access

nvidia-tuned/profiles/service/aks/nvidia-h100-inference.conf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@ include=nvidia-h100-performance
33
summary=Optimized for inference workloads (AKS-compatible)
44

55
[bootloader]
6-
# Isolate CPUs for inference processes, 2 per socket
7-
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}
86
# Allocate hugepages for better memory access
97
cmdline_hugepages=hugepagesz=2M hugepages=8192
108

nvidia-tuned/profiles/service/eks/nvidia-gb200-inference.conf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@ include=nvidia-gb200-performance
33
summary=Optimized for inference workloads (AWS-compatible)
44

55
[bootloader]
6-
# Isolate CPUs for inference processes, 2 per socket
7-
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}
86
# Allocate hugepages for better memory access
97
cmdline_hugepages=hugepagesz=2M hugepages=8192
108

nvidia-tuned/profiles/service/eks/nvidia-h100-inference.conf

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,6 @@ include=nvidia-h100-performance
33
summary=Optimized for inference workloads (AWS-compatible)
44

55
[bootloader]
6-
# Isolate CPUs for inference processes, 2 per socket
7-
cmdline_isolcpus=isolcpus=${f:cpulist_invert:${f:calc_isolated_cores:2}}
86
# Allocate hugepages for better memory access
97
cmdline_hugepages=hugepagesz=2M hugepages=8192
108

0 commit comments

Comments
 (0)