VLLM Inference Optimizations

[wlm64]

Hi All,

I was curious as to what optimizations VLLM deploys to improve the performance of LLM inference compared to a native pytorch call.

From reading the docs I saw paged attention and prefix caching as two primary optimizations but there were others such as disaggregated prefilling.

Assuming I simply call a model:

model = LLM(model=model_name, tensor_parallel_size=8)
outputs = model.generate(prompt, sampling_params)

What optimizations are automatically applied? I'm assuming paged attention and prefix catching but is there documentation discussing other optimizations? Furthermore I noticed that a fair amount of computation/time is done when you initially load the model prior to calling generate, this doesn't seem to be just loading weights either. Is there some precalculation of the KV Cache or something done here? Would appreciate any insight.

Thanks!

[lxcxjxhx]

When you call LLM(model=model_name, tensor_parallel_size=8).generate(...), vLLM automatically applies several optimizations:

Core optimizations (always enabled):

• PagedAttention: The KV cache is managed in pages (like virtual memory), reducing fragmentation and allowing dynamic allocation
• Continuous batching: Requests are batched dynamically as they arrive, maximizing GPU utilization
• CUDA kernels: Optimized attention kernels (FlashAttention, FlashInfer) for faster computation
• Tensor parallelism: Your tensor_parallel_size=8 splits the model across 8 GPUs with optimized communication

Additional optimizations (depend on config):

• Prefix caching: If enabled (enable_prefix_caching=True), common prompt prefixes share KV cache
• Chunked prefill: Long prompts are processed in chunks to avoid memory spikes
• Speculative decoding: If configured with a draft model, can speed up generation

About the initial loading time:
When you create the LLM object, vLLM does more than just load weights:

1. Model architecture initialization
2. Weight loading and dtype conversion (e.g., to FP16/BF16)
3. KV cache allocation based on gpu_memory_utilization
4. CUDA graph capture (if enabled) for faster execution
5. Warmup runs to initialize CUDA contexts

The KV cache is pre-allocated during initialization, not pre-calculated. The actual KV values are computed during the first forward pass.

Documentation:

• vLLM performance optimization guide
• vLLM architecture overview