The Workload Variant Autoscaler (WVA) is a Kubernetes-based global autoscaler for inference model servers serving LLMs. WVA works alongside the standard Kubernetes HPA and external autoscalers like KEDA to drive the scale subresource of inference deployments. The high-level details of the algorithms are documented here. It determines optimal replica counts for a given request traffic load by considering constraints such as GPU availability, energy budget, and performance budget (latency/throughput).
WVA introduces the concept of variants — multiple model servers in an InferencePool that all serve the same base model but differ in hardware configuration (e.g., GPU type), serving configuration (e.g., tensor parallelism, max batch size, quantization), or both.
Use cases include:
- P/D disaggregation: prefill is one variant, decode is another — variant = role in a disaggregated pipeline.
- batch-gateway: variants distinguish batch vs. interactive workloads sharing the same pool.
- Autoscaler: a costed serving configuration the autoscaler chooses among.
- Intelligent Autoscaling: Optimizes replica count by observing the current state of the system
- Cost Optimization: Minimizes infrastructure costs by picking the correct accelerator variant
See the architecture and autoscaling design docs for high-level algorithm details.
See the docs directory for design docs, developer guide, and more.
Prerequisites: deploy llm-d infrastructure (model servers) and create an HPA or KEDA object targeting each deployment.
WVA then:
- Continuously monitors request rates and server performance via Prometheus metrics
- Capacity model obtains KV cache utilization and queue depth to determine desired replica counts
- Actuator emits optimization metrics to Prometheus
- External autoscaler (
HPA/KEDA) reads the metrics and scales the deployment accordingly
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: llama-8b-autoscaler
namespace: llm-inference
annotations:
llm-d.ai/managed: "true" # Opt-in to WVA management
llm-d.ai/variant-cost: "10.0" # Optional, defaults to "10.0"
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: llama-8b
# minReplicas: 0 # scale to zero - alpha feature
maxReplicas: 2
behavior:
scaleUp:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 10
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 60
policies:
- type: Pods
value: 10
periodSeconds: 15
metrics:
- type: External
external:
metric:
name: wva_desired_replicas
selector:
matchLabels:
variant_name: llama-8b
exported_namespace: llm-inference
target:
type: AverageValue
averageValue: "1"More examples in config/samples/hpa/ and config/samples/keda/.
Behavioral change. The default saturation analyzer changes from V1
(percentage/spare-capacity-based) to V2 (token/capacity-based). The shipped
default entry in the saturation ConfigMap now includes an analyzers: section,
which selects V2. No code change or image rebuild is involved — analyzer selection
is driven entirely by config.
V2 may produce different scaling decisions than V1 for the same workload. Review your dashboards and alert thresholds after upgrading.
Staying on V1 (opt-out). Remove the analyzers: section (and the V2-only
scaleUpThreshold / scaleDownBoundary fields) from the default entry of your
saturation ConfigMap. The remaining kvCacheThreshold, queueLengthThreshold,
kvSpareTrigger, and queueSpareTrigger fields drive V1:
data:
default: |
kvCacheThreshold: 0.80
queueLengthThreshold: 5
kvSpareTrigger: 0.1
queueSpareTrigger: 3Apply with kubectl apply -f deploy/configmap-saturation-scaling.yaml; the change
takes effect immediately (the controller watches the ConfigMap).
V1 is deprecated and scheduled for removal in a future release. See the saturation scaling configuration guide for threshold ownership (which fields each analyzer reads) and migration details.
We welcome contributions! See CONTRIBUTING.md for guidelines.
Join the llm-d autoscaling community meetings to get involved.
Apache 2.0 - see LICENSE for details.
