Commit 9581cbb
authored
fix(nvidia-tuned): remove isolcpus from all inference profiles (#96)
## Summary
- Removes `cmdline_isolcpus` from all inference profiles (os/common h100
+ gb200, service/aks h100, service/eks h100 + gb200).
- Cleans up stale "Without cpu isolation due to tuned package version"
summary text in debian/11 and ubuntu/22.04 os-specific gb200 profiles.
- Bumps `package_version` to `0.3.2`.
## Background
Closes #95. `isolcpus=0,1,48,49` on AKS H100 nodes was isolating the
housekeeping cores (including CPU0, the default IRQ target) rather than
workload cores. Under high-concurrency inference serving this starved
network/softirq processing:
| Configuration | Throughput | TTFT p99 | Verdict |
|---|---|---|---|
| Full profile (with `isolcpus`) | 50,404 tok/s | 6,034 ms | FAIL |
| Same profile, `isolcpus` removed | 148,344 tok/s | 614 ms | PASS |
| Untuned stock AKS node | 146,452 tok/s | 547 ms | PASS |
Single-variable bisect from #95 confirmed `isolcpus` was the sole cause.
The remaining profile settings (hugepages, iommu=pt, pci=realloc=off,
init_on_alloc=0, vm.swappiness=1) are innocent.
## Test plan
- [ ] CI validate + dev build passes for `nvidia-tuned`
- [ ] `make test-package PACKAGE=nvidia-tuned` passes locally
- [ ] Verify no `isolcpus` in any tuned.conf: `grep -r isolcpus
nvidia-tuned/`
Signed-off-by: Alex Yuskauskas <ayuskauskas@nvidia.com>1 parent faf01e4 commit 9581cbb
8 files changed
Lines changed: 3 additions & 13 deletions
File tree
- nvidia-tuned
- profiles
- os
- common
- nvidia-gb200-inference
- nvidia-h100-inference
- debian/11/nvidia-gb200-inference
- ubuntu/22.04/nvidia-gb200-inference
- service
- aks
- eks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| |||
Lines changed: 0 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | 6 | | |
9 | 7 | | |
10 | 8 | | |
| |||
Lines changed: 0 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | 6 | | |
9 | 7 | | |
10 | 8 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| |||
Lines changed: 0 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | 6 | | |
9 | 7 | | |
10 | 8 | | |
| |||
Lines changed: 0 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | 6 | | |
9 | 7 | | |
10 | 8 | | |
| |||
Lines changed: 0 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
7 | | - | |
8 | 6 | | |
9 | 7 | | |
10 | 8 | | |
| |||
0 commit comments