[RFC]: Per-request activation of multiple LoRA adapters with static scales

[KisuYang]

Motivation.

vLLM's multi-LoRA serving currently routes each request to exactly one adapter: LoRARequest holds a single lora_name/lora_int_id, and the Punica metadata maps each token to one LoRA index. There is no way to apply two or more already-loaded adapters to the same request.

This blocks several practical workflows:

1. Composed / incrementally-trained adapters. A common pattern is training adapter_1 on top of a frozen adapter_0 (continual or staged fine-tuning). Serving the result requires both deltas active at once: y = Wx + s_0·B_0A_0x + s_1·B_1A_1x.
2. Combinatorial merge explosion. The current workaround is merging adapters offline (e.g. PEFT add_weighted_adapter(combination_type="cat")). With N adapters and per-request combinations, this requires materializing and loading up to N-choose-k merged artifacts, versus keeping only N adapters loaded with runtime composition.
3. Per-request scaling. Requests that weight the same adapters differently (0.7·style + 0.5·domain) cannot be served by any finite set of offline merges.

Related work:

• [RFC]: Multi-LoRA Composition for Diffusion Models vllm-omni#2149 proposes exactly this capability for diffusion models, where multi-LoRA composition with independent scales is a standard workflow (Diffusers set_adapters(names, weights)). This RFC is the LLM-side counterpart.
• [RFC]: Support router-driven mixtures of multiple LoRA adapters #49705 proposes router-driven adapter mixtures (X-LoRA/MixLoRA-style), where gate weights are computed per-token/per-layer by a learned router. This RFC is deliberately narrower: the adapter set and scales are static per request and supplied by the user, with no router, no per-token weights, and no new model components. The infrastructure introduced here (multiple active adapters per request, k-slot token→LoRA mapping, weighted accumulation) is a natural first stage that [RFC]: Support router-driven mixtures of multiple LoRA adapters #49705 could build on, since a router-driven mixture reduces to this proposal when gate weights are constant across tokens.

Proposed Change.

Semantics. For a request with adapters [(a_0, s_0), ..., (a_{k-1}, s_{k-1})], the output of each LoRA-targeted linear layer is base(x) + Σ_i s_i · (α_i/r_i) · B_i A_i x. Each adapter stays a first-class, independently-loaded entity at runtime; nothing is merged or materialized per combination.

Staged implementation:

1. API: extend LoRARequest to accept multiple adapters with optional per-adapter scales, keeping the single-adapter form backward compatible (either a new LoRACompositeRequest or an adapters: list[tuple[LoRARequest, float]] field). Offline LLM.generate() support first; OpenAI-compatible server support as a follow-up PR (e.g. request-level extra body "lora_adapters": [{"name": ..., "scale": ...}], since the model field is a single string).
2. Runtime metadata: generalize the token→LoRA mapping in the Punica wrapper from 1 slot to k slots per token. Tokens/requests using fewer than k adapters fill remaining slots with the sentinel.
3. Execution (reference implementation): invoke the existing lora_shrink/lora_expand path once per slot, accumulating into the output buffer with the per-adapter scale; the kernels' no-LoRA sentinel index (-1) already handles tokens using fewer than k adapters. Requests using a single adapter take slot 0 only and see no overhead; multi-adapter requests pay k× the (small) LoRA overhead. This validates the semantics with zero kernel changes.
4. Execution (optimized): once semantics are validated, extend the shrink/expand kernels to iterate the k slots in-kernel, amortizing launch overhead and intermediate-buffer traffic into a single fused pass. This is the intended end state; stage 3 exists so correctness and API can land without blocking on kernel work (same staging philosophy as [RFC]: Support router-driven mixtures of multiple LoRA adapters #49705).
5. Bounds: new engine arg max_loras_per_request (default 1 = today's behavior) so metadata buffers and scheduling stay bounded, mirroring the max_simultaneous_loras idea in vllm-omni#2149.
6. Scheduler: count each distinct adapter of a request against max_loras when forming batches.

Alternative considered and rejected: fusing the requested adapters on the fly into a cached rank-concatenated virtual adapter. This keeps today's single-adapter runtime untouched, but the fused cache grows with the number of distinct (combination, scales) tuples (unbounded once scales are continuous per-request values), interacts poorly with max_lora_rank, and adds fuse latency on first use. Runtime composition with independently-loaded adapters is the principled shape, and is also what #49705 needs underneath.

Testing: for fixed scales, runtime composition is mathematically identical to an offline PEFT cat-merged adapter, which gives an exact correctness oracle: logits must match up to dtype accumulation order. Plus regression tests asserting zero overhead and identical results for single-adapter requests, stage-4 kernels tested against the stage-3 reference path, and scheduler tests for max_loras accounting.

Out of scope (this RFC): routers / learned gates / per-token weights (#49705), training, cross-base-model adapters, prompt-embedding interactions.

Feedback Period.

Two weeks. I intend to follow up with a draft PR implementing stages 1–3 for the offline API once the API shape has maintainer sign-off.

CC List.

Reviewers familiar with LoRA serving and the Punica wrapper would be appreciated (happy to be pointed at the right owners). cc authors of vllm-omni#2149 and #49705 for alignment.

Any Other Things.

The forward semantics are a fixed weighted sum of standard LoRA deltas, with no new model components, so the correctness surface is small and independently checkable against offline merges. The value of the feature is precisely that this composition happens at serving time over independently-loaded adapters: bounded memory in the number of adapters (not combinations), per-request scale control, and infrastructure that router-driven mixtures (#49705) can later build on.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.