[Blog] Keeping vLLM Fast Under CPU Pressure: An sched_ext Scheduler for GPU Inference - #300
[Blog] Keeping vLLM Fast Under CPU Pressure: An sched_ext Scheduler for GPU Inference#300ianchen0119 wants to merge 6 commits into
Conversation
|
Would you please help to review the PR? |
There was a problem hiding this comment.
Pull request overview
Adds a new ecosystem/community blog post documenting how CPU scheduling contention can bottleneck vLLM GPU inference, and how the Gthulhu sched_ext (SCX) scheduler + tiered policies can recover decode throughput under CPU pressure.
Changes:
- Introduces a new long-form post with a reproducible MicroK8s + GB10 walkthrough and benchmark methodology.
- Documents tiered Gthulhu policy setup (node-level + pod/thread-level) targeting GPU IRQ handling and vLLM’s
EngineCore. - Presents benchmark results comparing EEVDF baseline vs Gthulhu with/without policies.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…or GPU Inference Signed-off-by: Ian Chen <ychen.desl@gmail.com>
…, MicroK8s casing, and align decode drop wording Signed-off-by: Ian Chen <ychen.desl@gmail.com>
1f13242 to
b88eb9c
Compare
|
cc @vadiklyutiy @ZJY0516 @NickLucche @ZhanqiuHu @njhill @llx-08 @jiahanc for vis since you are mentioned in the blog post. |
- Generate chart_decode.svg and chart_ttft.svg with matplotlib - Replace Chart.js runtime charts with static SVG images - Fix TTFT averages in overall comparison table (A: ~336ms, B: ~126ms, C: ~169ms) - Correct misleading summary that implied C had lower TTFT than B Signed-off-by: Ian Chen <ychen.desl@gmail.com>
c9030b3 to
52246af
Compare
|
@simon-mo @youkaichao — would either of you be able to help review this ecosystem blog PR when you have time? It documents a reproducible vLLM-on-GB10 experiment showing how CPU contention affects decode throughput and how a The PR has been open since Aug 7 and is mergeable; a maintainer review on whether this is suitable for the vLLM blog (and any changes needed for publication) would be very helpful. Thanks! |
This PR adds a new community/ecosystem blog post about improving vLLM inference performance under CPU contention using Gthulhu, a Linux sched_ext (SCX) scheduler.
The post walks through a reproducible experiment on a DGX Spark (GB10, arm64) running vLLM (Qwen2.5-0.5B-Instruct) on MicroK8s, and shows that CPU scheduling — not GPU capacity — is the bottleneck when an inference node is under CPU pressure from noisy neighbors.