Disclaimer: none of this was generated by AI. These are my hands-on experience and study notes, brain dumps from digging deep into AI inference and how it changes the paradigm of running AI on Kubernetes (sorry it is a bit long). I’ve put together my thought in sessions that helped me to have a better understanding of the whole Inference workflow.

TL;DR: don't treat LLM workloads on Kubernetes like regular web apps. The defaults that work for stateless services (round-robin load balancing, CPU/memory autoscaling, default health checks, the device-plugin GPU model) all fail in different ways for inference. What you actually need for larger AI models on Kubernetes: DRA for GPU topology, LeaderWorkerSet + Kueue/Volcano when one replica spans multiple pods, the Kubernetes Gateway API Inference Extension for prefix-aware routing, KEDA scaling on KV cache pressure and queue depth (not CPU), and reliability patterns built around minute-long cold starts and stateful pods. This post walks through how all of those pieces fit together, using a Mistral Large 123B deployment as the running example.

If you're interested on learning how to perform AI Inferece Infrastructure upgrade without downtime on top of Kubernetes, please jump into zero downtime maintenance guide.

Contents

1. High-level overview of what Inference is
2. How to do Inference capacity planning?
3. Kubernetes & GPUs
4. How Kubernetes schedule LLM pods?
5. Kubernetes Node & Application Scaling
6. How to route requests intelligently?
7. Split prefill and decode
8. Observability
9. Reliability

I've been spending a lot of time lately learning and playing with AI inference since I got my Ray Foundations certification back in September, and it's been impressive (and humbling) to see all the challenges that come with running it at scale on Kubernetes. You pretty much have to throw away the mental model you use for web apps to understand how to properly set up inference for low latency, high throughput, and reasonable cost. LLM workloads are open-ended, every request has a different input and output size, and they're inherently stateful, since model weights live in GPU memory and cold starts are painfully slow.

At the beginning I was overwhelmed by all the new terms, matrix math, transformer architecture, KV caches, tensor and pipeline parallelism. It's a wild journey, but worth it. It's kept me motivated to learn a bunch of things I didn't even know existed a few months ago.

In this post I'll share my thoughts coming from an SRE / platform / large-scale Kubernetes background on what you need to consider before and during the design of an AI inference service. Running Ollama locally on your laptop is a very different beast from serving a 671B model in production with thousands of concurrent users.

High-level overview of what Inference is

Inference is the process of making the LLM generate outputs based on what it was trained on. Basically, inference is when the user is actually using the model, it's actively generating tokens to answer the user's request. In the LLM world, training and inference are the two most important phases, and inference is the more critical of the two from an operations standpoint, since your end users interact with it directly.

LLMs today (May 2026) are based on the transformer architecture, which is a complex beast. For the sake of this post I'll keep it very high level and focus only on the inference side. In a transformer, when generating a new token, the attention mechanism needs to look at every previous token to decide what to output next. Each token has three associated vectors: Q (query), K (key), and V (value). For the new token being generated, its Q gets compared against the K of every previous token to figure out which past tokens are relevant, then those V vectors get mixed together to inform the output.

Inference has two main phases with very distinct characteristics: prefill and decode. Prefill is when you send a prompt to the LLM, the inference server CPU tokenizes the prompt, generates token-ids, assembles the tensors, allocates memory buffers, and notifies the GPU it has data to process. The GPU then handles prefill, which processes the entire input in parallel, all tokens at once, and stores the resulting K and V vectors in the KV cache. This phase is very compute-bound. You can think about prefill as "processing the prompt," and this is where TTFT (Time To First Token) comes from.

The KV cache is used to store all the K/V pairs. Without it, every time a new token is generated you'd have to recompute K and V for every previous token in the context, which is a waste of compute and money. Instead, you compute K and V once and store them in HBM (GPU memory). When new tokens are generated during the decode phase, you compute K and V only for the new token and reuse everything that was already computed, which directly helps with latency.

The decode phase generates output tokens one at a time. For each new token it reads the entire KV cache from HBM, runs the matrix multiplications, appends the new token's K/V to the cache, outputs one token, and repeats. This operation is very memory-bound, since you're reading gigabytes of KV cache per generated token, and this is why decode speed scales with HBM bandwidth, not raw compute.

Before jumping into Kubernetes specifics, let's first understand what the model we want to host looks like, so we have the information we need to do proper capacity planning. In this post I'm using Kubernetes with vLLM as the inference framework.

How to do Inference capacity planning?

To do capacity planning for your AI inference setup, you first need to understand the characteristics of the model you're going to serve. How many parameters does it have? Which precision does it use? What's the size of the context window? What's the average tokens per request?

Let's take mistralai/Mistral-Large-Instruct-2411: 123B parameters, 128K context window, BF16 precision (2 bytes per parameter). Right off the bat you can say you need 246 GB of GPU memory just to host the weights (123B × 2 bytes).

For inference, GPU memory has to fit three things: model weights, activation overhead, and the KV cache. We know weights are 246 GB. Add ~30 GB for activations and CUDA overhead, so call it ~280 GB. But what about the KV cache? Remember, the KV cache is the space in HBM where prefill stores processed tokens and decode reads from and appends to, it needs enough room to handle all in-flight requests.

There's a whole pile of math to compute KV cache size exactly. You need things like number of layers, number of KV heads, and head dimension, all of which you can find in the model's config.json. On the hardware side, some of options you have today are: NVIDIA H100 (80 GB HBM3), H200 (141 GB HBM3e), B200 (192 GB HBM3e). A natural configuration for this model is 8× H100 on a single node, giving you 640 GB of HBM connected over NVLink.

280 GB for weights + overhead leaves you ~360 GB for the KV cache. You want enough KV cache headroom for batched generation to actually scale, that's what lets you handle more concurrent requests and longer contexts.

Other possible hardware setups:

• 4× H200 (564 GB)
• 2× B200 (384 GB)
• 4× B200 (768 GB)

Now that we know weights + KV cache budget, we can estimate how many requests this setup will actually handle.

For Mistral Large specifically, each token takes about 0.344 MB of KV cache (at BF16). At the full 128K context, a single request consumes 0.344 MB × 128,000 ~ 44 GB of KV cache. With 360 GB of KV cache budget, that's ~8 concurrent requests at maximum context, worst case and pretty unrealistic, since not everyone is going to push 128K tokens at once.

A more realistic average of 8K context per request: 0.344 MB × 8,000 ~ 2.75 GB per request, so 360 GB / 2.75 GB ~130 concurrent requests. At 2K average context, you're looking at ~520 concurrent requests.

One important note, these are upper-bound numbers assuming static allocation. vLLM uses PagedAttention to manage KV cache as fixed-size blocks, which means it can pack requests more efficiently than naive math suggests. You can also enable FP8 KV cache, which roughly halves the per-token KV cost in exchange for a small quality hit, worth knowing about when you're memory-constrained.

If you don't already have these numbers from your real workload, before going further with capacity planning make sure you have observability in place. On Kubernetes that means the NVIDIA DCGM Exporter scraped by Prometheus to collect GPU metrics (memory usage, SM utilization, NVLink bandwidth, power, etc.). Observability has to be a first-class citizen of your AI platform, without it, capacity planning is guesswork.

To summarize, to host Mistral Large Instruct 2411:

• We need 8× H100 (80 GB) on a single node, connected via NVLink
• At 8K avg context, we can handle ~130 concurrent requests (one request = one inference call)
• At 2K avg context, we can handle ~520 concurrent requests

A note on splitting models: Mistral Large at 246 GB doesn't come close to fitting on a single GPU, so we have no choice, we're splitting it across the 8 GPUs of one node using tensor parallelism. The complexity this adds is topology awareness: the 8 GPUs need to be NVLink-connected so the all-reduce ops between them aren't bottlenecked, which is what DRA solves for us. Gang scheduling isn't needed here, because the whole replica still fits in a single pod, TP happens inside the pod, not across pods. Gang scheduling only enters the picture if a single replica spans multiple pods (e.g. pipeline parallelism across nodes, or disaggregated prefill/decode). For a model like this on 8× H100 in one box, keep it simple: one pod, tensor parallelism, NVLink topology via DRA.

Kubernetes & GPUs

The common setup today is to install the NVIDIA GPU Operator, which deploys the Device Plugin as a DaemonSet on your GPU nodes. The Device Plugin talks to the kubelet via gRPC and advertises GPUs as an extended resource (nvidia.com/gpu). Kubelet then exposes that capacity to the scheduler, which uses it to place pods. This is the most widely used flow today.

The main limitation of this flow is that the kubelet only sees GPUs as opaque counters. Inside the kubelet there's a component called the Topology Manager that coordinates hints from the Device Manager (which talks to the Device Plugin), CPU Manager, and Memory Manager, and tries to align allocations on the same NUMA node. The problem is twofold:

1. Its topology policy (none, best-effort, restricted, single-numa-node) is set per kubelet, so it applies uniformly to every pod on that node, you can't say "this pod needs strict alignment, that one doesn't."
2. It's NUMA-aware only. It has no understanding of NVLink domains, NVSwitch fabrics, or PCIe switch topology.

So with the Device Plugin path, you can't actually request things like "I want 4 GPUs connected via the same NVLink domain" or "I want the GPU on the same PCIe root complex as my NIC”, and that kind of locality matters a lot when you're doing inference at scale with tensor parallelism, since the all-reduce between GPUs goes over those interconnects.

The solution is Dynamic Resource Allocation (or DRA), which went GA in Kubernetes 1.34. DRA replaces the count-based device-plugin model with an attribute-aware one: the NVIDIA DRA driver publishes detailed ResourceSlice objects describing every GPU's properties (model, memory, NVLink domain, MIG capability, PCIe location, etc.), and workloads describe what they need via ResourceClaim / ResourceClaimTemplate.

The scheduler then matches claims against the live device inventory using CEL expressions, and binds the chosen devices atomically with the pod. This gives you per-workload, attribute-aware allocation at the device level, exactly what you need for "4 GPUs on the same NVLink domain" type constraints.

Note: If you're not on Kubernetes 1.34 yet, start planning the upgrade. The H100 has ~3.35 TB/s of HBM bandwidth and ~900 GB/s of NVLink bandwidth between GPUs, when you split a model with tensor parallelism, the speed of the inter-GPU links directly impacts inference latency. Making the scheduler topology-aware is not an optimization, it's the difference between NVLink (fast path) and falling back to PCIe (much slower).

On 1.34, DRA is enabled by default. You still need the NVIDIA GPU Operator, but you configure it to deploy the NVIDIA DRA driver (k8s-dra-driver-gpu) instead of (or alongside) the classic Device Plugin. From there, your workload references a ResourceClaimTemplate instead of resources.limits: nvidia.com/gpu: N, and the DRA scheduler plugin participates in the scheduling decision, evaluating which nodes have devices that satisfy the claim and allocating those specific devices atomically with the pod binding. In short: DRA doesn't run after the scheduler admits a pod (that took me a while to understand), it runs as part of the scheduling decision, giving the scheduler the topology vocabulary the Device Plugin path never had.

How Kubernetes schedule LLM pods?

The default behaviour of Kubernetes is to use the kube-scheduler to place pods on nodes. In a nutshell, it filters nodes that have enough free resources to host the pod, scores the survivors, and the node with the highest score wins. For LLM inference, this default behaviour breaks down in two distinct scenarios, and they're often conflated, so let's separate them clearly.

Scenario 1: One pod, multiple GPUs on the same node This is what we have with the Mistral Large setup above: one pod requesting 8 GPUs on a single node, using tensor parallelism. The kube-scheduler can place this fine count-wise, but it doesn't know that all 8 GPUs need to be on the same NVLink domain for TP all-reduces to not become a bottleneck. This is a topology problem, not a scheduling-coordination problem. The fix is DRA (previously discussed) it gives the scheduler the vocabulary to understand "all 8 GPUs must share an NVLink fabric." No gang scheduling needed, no extra controllers, just smarter resource claims.

Scenario 2: One replica, multiple pods (across nodes) This shows up when the model doesn't fit on a single node, think 405B, DeepSeek-R1 671B, or very long-context models where even the KV cache pushes you across nodes. It also shows up in disaggregated serving, where prefill and decode run on separate pods that must come up together to form one logical replica.

Now you have a problem the default scheduler genuinely can't solve. If you need 5 pods for one replica, kube-scheduler will try to place them one by one, and if only 3 can fit, you end up with 3 idle pods waiting for the other 2, holding expensive GPUs hostage while the replica is non-functional. This is the gang scheduling problem: schedule all N pods or none of them, atomically. There are two layers you need for this scenario:

1. A workload API that expresses "one replica = N pods with distinct roles": A Deployment can't do this, it manages identical, interchangeable pods. A StatefulSet gets closer but still treats pods as homogeneous. The right primitive is LeaderWorkerSet (LWS), a SIG-driven Kubernetes API designed specifically for multi-host inference. You define one leader template and one worker template, set a group size (e.g. 2 pods per replica), and LWS treats the leader+workers as a single scalable unit. This is what vLLM's multi-node deployment guide actually uses, the leader runs the Ray head, the workers join the Ray cluster, and they load shards of the model together.

2. An admission controller that gang-schedules the group: Even with LWS, you still need something to ensure all pods in a group are admitted atomically. This is where Kueue or Volcano come in, they sit in front of kube-scheduler and either release all pods at once for scheduling or hold them in a queue until capacity is available. Without this layer, LWS would still let kube-scheduler partially admit a group and you'd be back to the half-dead-replica problem.

Where Kueue earns its keep beyond gang scheduling If you're running the same Kubernetes cluster for both training and inference, Kueue does a lot more than just gang admission:

• Quota management between teams or workload classes (production inference, batch inference, training, dev)
• Priority and preemption, when inference load spikes, you can preempt lower-priority training jobs to free up GPUs
• Workload queueing when capacity is exhausted, with deterministic ordering instead of a chaotic "whoever retries fastest wins"
• DRA-aware quotas, so "team A has 32 H100s" actually accounts for DRA-claimed devices

Volcano covers similar territory and has a longer history in HPC/batch use cases. Kueue is the more Kubernetes-native option and is what most modern AI platforms are converging on.

Putting it together The full picture of what you need depends on the topology of your workload: Workload shape What you need Model fits in 1 GPU Deployment + kube-scheduler Model fits in 1 node, multi-GPU (TP only) Deployment + DRA Model spans multiple nodes (TP+PP, disaggregated) LWS + DRA + Kueue/Volcano (gang scheduling) Shared cluster across teams or with training Add Kueue regardless of model size For our Mistral Large example, we're firmly in row 2 8× H100 on one node, one pod, tensor parallelism, DRA for NVLink topology. No LWS, no gang scheduling. Design for the more complex case in your platform but don't implement it until you actually have a model that demands it.

Kubernetes Node & Application Scaling

You'll want to automatically scale your Kubernetes nodes when traffic spikes and you need to serve more concurrent requests. The goal is simple: a new node joins the cluster, gets a pod scheduled on it, and starts serving traffic as fast as possible. The two common options are Cluster Autoscaler (CAS) and Karpenter.

Cluster Autoscaler is the original. When the scheduler can't place a pending pod, CAS waits for a configurable interval and then scales up the matching node group by incrementing the desired count of the underlying autoscaling group (ASG on AWS, MIG on GCP). The cloud provider provisions the instance, it joins the cluster, and pods get scheduled on it. It works, lots of companies run it in production, and it's well understood.

Karpenter is the more modern option, and it's worth seriously considering for GPU workloads. A few reasons that I learned:

• Faster provisioning: Karpenter talks directly to the EC2 Fleet API (on AWS) instead of going through ASGs, which skips a layer of indirection and tends to shave meaningful time off cold starts. When your "cold start" already includes pulling a 50GB+ container image and loading 246GB of model weights into HBM, every second of node-provisioning latency matters.
• Workload-aware instance selection: Instead of pre-defining static node groups, Karpenter looks at the pending pod's actual requirements (CPU, memory, GPU type, zone, architecture, etc.) and picks the cheapest instance type that fits. For GPU clusters where you might want H100s for one workload and L4s for another, this is a real operational simplification.
• Faster consolidation: When demand drops, Karpenter is more aggressive about bin-packing pods onto fewer nodes and terminating the empty ones.

What's specific to GPU node scaling A few things bit me on GPU clusters that don't matter as much for general workloads:

1. Cold starts are dominated by model loading, not node provisioning: Even with Karpenter giving you a node in 30 seconds, you then have to pull the container image (which for vLLM with CUDA libraries is typically 5–15 GB) and load model weights into GPU memory. For Mistral Large that's 246 GB of weights moving from somewhere (S3, a model registry, a network volume) into HBM. If you're pulling from Hugging Face on cold start, you're looking at minutes, not seconds. Real solutions include:

• Baking model weights into a custom AMI or container image (fastest, but inflexible)
• Using a shared filesystem like FSx for Lustre, EFS with high throughput mode, or a model cache layer
• Pre-warming nodes with the image already pulled
• Pre-loading weights into a tmpfs or local NVMe and mounting it into the pod

2. GPU capacity isn't always available on demand: H100 and B200 capacity on cloud providers is genuinely constrained. You can ask Karpenter for an 8× H100 instance and get a InsufficientInstanceCapacity error. For predictable production traffic, Capacity Blocks (AWS) or Reservations (GCP/Azure) are often a better fit than pure on-demand scaling. Karpenter supports capacity reservations as a first-class concept.

3. Spot is risky for inference but tempting: Spot H100s can be ~70% cheaper, but a spot reclaim during a generation is bad UX, the user's request dies mid-stream. Spot is reasonable for batch inference or for training, but for online serving most people stick with on-demand or reservations. If you do use spot, configure aggressive PDBs and graceful shutdown so in-flight requests can drain (but be careful).

4. Pre-provisioned warm pools: For latency-sensitive inference, scaling reactively (pending pod → new node → image pull → weight load) is often too slow. A common pattern is to keep a warm pool of "ready" nodes, image pulled, model loaded, that can take a pod immediately. Karpenter doesn't natively do warm pools, but you can fake it by running low-priority placeholder pods that get evicted when real workloads land.

5. Scale signals matter more than the autoscaler: Cluster autoscaling only reacts to pending pods, which means your pod autoscaler (HPA/KEDA) needs to be triggering replica scale-up before the cluster is saturated. For LLM workloads, GPU utilization is a misleading metric (a model loaded into HBM shows high "utilization" even when idle). Better signals: requests-per-second per replica, queue depth (vLLM exposes vllm:num_requests_waiting), TTFT/ITL (Inter-Token Latency) percentiles, or KV cache utilization. KEDA with a Prometheus scaler hitting vLLM's metrics endpoint is the typical pattern.

From a inference server scale perspective, the usual signals, CPU, memory, request count, don't work for LLM inference. The signals that actually matter are KV cache pressure (vllm:gpu_cache_usage_perc) and queue depth (vllm:num_requests_waiting), backed by SLO-level metrics like p95 TTFT and ITL. KV cache pressure tells you the replica is running out of memory to admit new requests; queue depth tells you it's already saturated. Scale before either becomes critical, KV cache at 90% is too late, queue depth > 0 sustained means you're already adding latency. Pick thresholds (e.g. 70–80% KV usage) that give new replicas time to come up, since cold start for LLM pods is dominated by model loading and typically measured in minutes, not seconds. At the pod level, KEDA with a Prometheus scaler pointed at vLLM's /metrics endpoint is the standard pattern. More on the specific metrics later.

How to route requests intelligently?

First thing: don't put a round-robin load balancer in front of your LLM service. Inference requests are not regular HTTP requests, and treating them as such will tank your latency and your GPU utilization.The reason is the KV cache. If a user asks the chatbot a question and then sends a follow-up, the second request shares a huge chunk of context (system prompt, conversation history, any RAG documents) with the first. If both requests hit the same pod, vLLM's prefix caching reuses the already-computed KV cache for the shared prefix and only computes K/V for the new tokens. If they hit different pods, the second pod has to recompute the entire prefix from scratch, a full prefill on potentially thousands of tokens, which directly hits your TTFT and burns compute.The right solution is the Kubernetes Gateway API Inference Extension (often shortened to "Inference Gateway" or GIE). The architecture looks like this:

Client → Envoy + EPP (Endpoint Picker) → vLLM pods
                    ↑
              KV cache indexer

The Endpoint Picker (EPP) is the brain. It runs as a service alongside Envoy and makes the routing decision per request based on a three-stage pipeline: Filter: drop pods that are overloaded, unhealthy, or otherwise ineligible (e.g. KV cache near capacity, request queue too deep) Score: score the survivors based on signals like prefix cache hit probability, current queue depth, and load Select: pick the best pod and route the request there

The EPP relies on a KV cache indexer, a component that tracks which prefixes are cached on which pods, so it can route a new request to a pod that already has the relevant prefix in HBM. This is what gives you prefix-aware routing across the fleet, not just within a single pod. NVIDIA Dynamo uses these same primitives (with its own KV-aware router), the Inference Gateway is the open, Kubernetes-native version of the same idea. A few important notes:

• It's not strictly session affinity. Session affinity (sticky sessions by cookie or header) is a simpler concept and the gateway can do it, but the real value is prefix-aware routing, routing based on actual cached content, not just a sticky identifier. Two users hitting the same system prompt benefit from landing on the same pod even though they have no session relationship. Conversely, the same user's two requests might route to different pods if their prefixes have diverged enough that another pod is actually a better fit.
• It handles load and prefix together. Pure prefix-affinity would happily overload a single pod if everyone shares a prefix. The scoring stage balances cache locality against current load, which is the whole point of having a smart router instead of a hashing rule.
• It's model-aware. The gateway knows about model names, LoRA adapters, and inference-specific concepts that a generic L7 load balancer doesn't. If you have multi-LoRA serving, it can route based on which pod has which adapter loaded.

Split prefill and decode

As mentioned earlier, prefill and decode have very different hardware characteristics. Prefill is compute-bound, it processes the whole prompt in parallel and hammers the GPU's tensor cores. Decode is memory-bandwidth-bound , it generates one token at a time and spends most of its time reading the KV cache from HBM.

Running both on the same GPU means they step on each other: a long prefill blocks decodes from running, and decodes underutilize the compute that prefill needs. Disaggregated serving puts prefill and decode on separate pods. Prefill pods handle prompt processing, decode pods handle token generation, and the KV cache produced by prefill is transferred to the decode pod when it's done. When is this worth it?

• High request volume: For a handful of concurrent requests, don't bother, a unified pod with vLLM's chunked prefill handles the interleaving fine. Disaggregation starts mattering at hundreds of concurrent requests when prefill/decode contention shows up in your TTFT.
• Long prompts: The longer the context, the more prefill dominates and the more it pays to isolate it. RAG and agentic workloads are the natural fit.
• Tight TTFT SLOs: If you need fast first tokens, splitting lets you scale prefill capacity independently.

The interesting capability they offer is per-request routing: short prompts go to a unified pod, long prompts go to the disaggregated path. You don't have to commit the whole fleet to one mode. ON the Kubernetes side, this is where everything from earlier sections clicks together: LWS to express the multi-pod replica, Kueue/Volcano for gang scheduling, DRA for topology (the KV transfer is bandwidth-sensitive), and the Inference Gateway to route based on which pods are prefill vs. decode.

My take: Don't start here. Build the unified-pod path first, get observability in place, and only reach for disaggregation when your metrics show prefill/decode contention is actually the bottleneck. Design for it in your architecture so you can adopt it later, but for the Mistral Large example with ~130 concurrent requests, you don't need it. For thousands of concurrent requests on long contexts, you probably will.

Observability

A handful of metrics matter for inference SRE , and most of them aren't the ones you'd reach for from a web-app background. Forget CPU, request count, and GPU utilization (the last one is genuinely misleading for LLMs). Focus on these:

• TTFT (Time To First Token): your primary latency indicator. Climbing TTFT means prefill is contended or you're missing the prefix cache. Track p50, p95, p99 per replica.
• TPOT / ITL (Time Per Output Token / Inter-Token Latency): your generation speed. Spikes here mean decode is contended, usually because KV cache pressure is forcing evictions or recomputation.
• KV cache utilization (vllm:gpu_cache_usage_perc): the leading indicator of replica saturation. Sustained above ~80–85% means you're about to start hurting; this should be a scaling signal, not just an alert.
• Queue depth (vllm:num_requests_waiting): anything sustained above zero means you're already adding latency. Another scaling signal.
• GPU health via DCGM Exporter: Xid errors, ECC errors, thermal throttling, NVLink status. At fleet scale these are routine; you need visibility.

Tie these together: TTFT + TPOT tell you whether SLOs are at risk, KV cache + queue depth tell you why and feed scaling decisions, DCGM tells you when hardware is the problem. Without all three layers you'll be scaling reactively or chasing the wrong cause.

One important aspect of observability for inference: avoid averages for latency. LLM latency distributions are heavily right-skewed, a single long prompt can take orders of magnitude longer to prefill than a typical one, and the mean gets dragged with it.

Imagine 10 requests with TTFTs of 100, 102, 100, 99, 104, 110, 90, 3000, 95, and 105ms. The average is ~390ms, which makes the service look much slower than it actually is for almost every user, one outlier (probably a very long prompt or a cache miss) dominates the math. Use percentiles instead. They tell you what a specific portion of your users actually experience:

• p50 (median), half your requests are faster than this, half are slower. Useful as a baseline but easy to be fooled by, since it ignores the tail.
• p95, 95% of requests are faster than this. This is the one to alert on for most inference SLOs.
• p99, your worst 1%. At scale this is still thousands of users per day having a bad experience. Track it, don't ignore it.

A quick note on GPU utilization. The nvidia-smi "utilization" number is misleading, it only tells you whether a kernel is running, not whether the GPU is doing useful work. The right metrics are MFU (Model FLOPs Utilization) for prefill, which is compute-bound, and MBU (Memory Bandwidth Utilization) for decode, which is memory-bound. On an H100, healthy MFU during prefill is ~40–50%, and healthy MBU during decode is ~60–80%. You won't put these on a primary dashboard, but they're the metrics you reach for when answering "am I using this GPU efficiently or burning money?"

One more thing worth saying, these metrics measure the health of the inference server, not the model. If your TTFT and TPOT look fine but users say the answers are bad, that's a model quality issue (prompt engineering, fine-tuning, model selection), not an inference infrastructure issue. The two get conflated all the time.

Reliability

LLM inference reliability looks different from web-app reliability. Cold starts are minutes, not seconds. Pods are stateful (KV cache is real state). Failures are expensive (one dead 8× H100 pod is ~$25/hour of capacity gone). And the work itself is non-uniform , a single bad request can degrade an entire replica. Here are the dimensions worth thinking through.

Health checks that actually mean something The default Kubernetes pattern of /healthz returning 200 if the process is up is misleading for inference. A vLLM pod can be "up" while:

• The model isn't loaded yet (cold start still in progress — 5+ minutes)
• The GPU is wedged but the HTTP server is still responding
• KV cache is exhausted and the pod is rejecting new requests
• CUDA threw an error and the engine is in a degraded state

You need three distinct probes:

• Startup probe: long timeout (10+ minutes), tolerates the model-loading phase. Without this, your readiness probe will fail repeatedly during cold start and Kubernetes will kill the pod before it ever serves a request.
• Readiness probe: checks whether the engine can actually accept work. Hit an endpoint that performs a real test, not just TCP open. vLLM's /health is a good start; for production, consider a synthetic generation that exercises the GPU path.
• Liveness probe: checks for wedged state. Generous timeouts here, you don't want to kill a pod mid-generation of a long response. A common pattern is to check that the engine has made forward progress recently (tokens generated in the last N seconds when there are active requests).

Graceful shutdown and request draining When a pod is terminated, in-flight generations need to finish or fail cleanly. Killing a pod mid-stream means the user sees a truncated response, which is much worse than a 503 they can retry. Make sure:

• terminationGracePeriodSeconds is long enough to drain (60–300s depending on max output length)
• vLLM stops accepting new requests immediately on SIGTERM but lets in-flight ones complete
• The Inference Gateway / EPP marks the pod as draining and stops routing to it
• For LWS-based multi-pod replicas, the whole group drains together, not pod-by-pod

This matters a lot during rolling upgrades, node consolidation, and spot reclaims.

Pod-level failures and replica isolation A single pod can degrade in ways that aren't obvious from outside:

• KV cache fragmentation causing throughput collapse
• CUDA OOM that doesn't kill the process but breaks the engine
• A specific prompt hitting an edge case in the model and hanging
• NCCL hangs in multi-GPU setups when one GPU misbehaves

Production patterns:

• Circuit breaker at the gateway. If a pod's error rate or p99 latency exceeds threshold, the EPP stops routing to it for a cool-down period. Hard requirement at scale.
• Per-request timeouts. Generation should have a hard ceiling (e.g. 5 minutes). A hung generation locks GPU resources indefinitely.
• Periodic pod recycling. Some teams restart pods on a schedule (e.g. every 24h) to avoid slow accumulation of fragmentation or memory leaks. Controversial, better to fix the underlying issue, but pragmatic.

Multi-pod replica failure modes (LWS) This is the failure mode most teams underestimate. When one replica is multiple pods (LWS-based multi-node inference), the failure semantics are different from a Deployment:

• If one worker dies, the whole replica is dead, the other workers are useless without it
• The leader's death also kills the replica
• Recovery means restarting the entire group, not just the failed pod
• Cold start is now the cold start of the slowest pod in the group

LWS supports this via restartPolicy: RecreateGroupOnPodRestart, which kills and restarts the whole group on any pod failure. Use it. The alternative, pod-level restart, leaves you with a broken replica that looks healthy.

Plan for replica-level redundancy: if a 4-pod replica takes 8 minutes to come back, you need enough other replicas to absorb the load during that window.

GPU and node failures Guess what? GPUs fail. At fleet scale, this is regular operations, not an emergency. What you need:

• NVIDIA DCGM exporting metrics to Prometheus
• Node Problem Detector with GPU plugins, surfacing hardware issues as node conditions
• Automatic node draining when a node enters a degraded state (cordoned + workloads rescheduled)
• GPU-aware PDBs so you don't drain the only remaining replica when you're already down a node

For multi-node training/inference (LWS), a single GPU failure on one worker takes out the entire replica. The blast radius is bigger than you'd think.

Cold start as a reliability problem Cold start isn't just a performance problem , it's a reliability problem. If your replica takes 8 minutes to come up and a node dies, you've got 8 minutes of degraded capacity. At 3 a.m. with a spot reclaim cascade, this can spiral. Mitigations:

• Pre-warmed replicas (warm pool of ready-to-serve pods)
• Model weights baked into the AMI or pre-loaded onto NVMe, not pulled from object storage at startup
• Headroom: run with enough replicas that losing one doesn't immediately breach SLOs while the replacement is coming up
• Surge capacity during deploys: don't drain old replicas until new ones are warm and serving