[Bug]: rsLoRA scaling factor ignored for MoE expert LoRA adapters (pack_moe / pack_moe_stacked recompute alpha/rank)

[kushaldabbe]

Your current environment

The output of python collect_env.py

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 26.04 LTS (x86_64)
GCC version                  : (Ubuntu 15.2.0-16ubuntu1) 15.2.0
Clang version                : Could not collect
CMake version                : version 4.2.3
Libc version                 : glibc-2.43

==============================
       PyTorch Info
==============================
PyTorch version              : 2.13.0+cpu
Is debug build               : False
CUDA used to build PyTorch   : Could not collect
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.14 (main, Sep  1 2026, 14:16:52) [Clang 22.1.3 ] (64-bit runtime)
Python platform              : Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.43

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen 7 4800H with Radeon Graphics
CPU family:                              23
Model:                                   96
Thread(s) per core:                      2
Core(s) per socket:                      8
Socket(s):                               1
Stepping:                                1
BogoMIPS:                                5788.91
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr virt_ssbd arat umip rdpid
Hypervisor vendor:                       Microsoft
Virtualization type:                     full
L1d cache:                               256 KiB (8 instances)
L1i cache:                               256 KiB (8 instances)
L2 cache:                                4 MiB (8 instances)
L3 cache:                                4 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.3.5
[pip3] pyzmq==27.2.0
[pip3] torch==2.13.0+cpu
[pip3] torchaudio==2.11.0+cpu
[pip3] torchcodec==0.16.0+cpu
[pip3] torchvision==0.28.0+cpu
[pip3] transformers==5.16.1
[pip3] triton==3.8.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.28.1rc0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled

==============================
     Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_kushal

🐛 Describe the bug

Adapters trained with rank-stabilized LoRA (use_rslora: true) are applied with the wrong scaling magnitude when they target MoE expert modules.

PEFTHelper.__post_init__ computes the per-adapter factor vllm_lora_scaling_factor = lora_alpha / sqrt(r) for use_rslora adapters (vllm/lora/peft_helper.py:56-60), and every LoRALayerWeights carries it (from_config receives it at vllm/lora/lora_weights.py:69). The dense/packed path honors it: PackedLoRALayerWeights.pack calls lora.optimize(), which merges the stored self.scaling into lora_b.

The MoE path does not. Both PackedLoRALayerWeights.pack_moe (vllm/lora/lora_weights.py:205) and pack_moe_stacked (:253) recompute scaling = lora_alpha / rank from the raw fields and ignore the stored factor. The real load chain for adapters loaded from disk reaches this recomputation: LoRAModel.from_lora_tensors -> from_config -> LoRAModelManager._create_merged_loras_inplace -> pack_moe (vllm/lora/model_manager.py:813).

Impact: for r=16, lora_alpha=32 the correct factor is 8, the applied factor is 2 — the adapter delta is sqrt(r) times weaker (4x here). No error is raised; outputs are silently wrong. rsLoRA support for dense layers was added in #6909; the MoE packing paths arrived later and never picked it up.

Reproduction (CPU only, no GPU/model/server needed; verified on main at f2e2936):

import torch
from vllm.lora.lora_weights import LoRALayerWeights, PackedLoRALayerWeights

RANK, ALPHA = 16, 32
FACTOR = ALPHA / RANK**0.5  # 8.0, as stored by PEFTHelper for use_rslora

def make(name):
    return LoRALayerWeights(
        module_name=name, rank=RANK, lora_alpha=ALPHA,
        lora_a=torch.randn(RANK, 4), lora_b=torch.randn(6, RANK),
        scaling=FACTOR,
    )

packed = PackedLoRALayerWeights.pack_moe(
    [make("experts.w1"), make("experts.w2"), make("experts.w3")], "experts"
)
print(packed.scaling)

Observed output:

[2.0, 2.0, 2.0]        # recomputed lora_alpha / rank

Expected output:

[8.0, 8.0, 8.0]        # the stored per-adapter vllm_lora_scaling_factor

pack_moe_stacked shows the same recomputation with stacked tensors.

Expected behavior: both MoE packers should use the stored per-adapter scaling, matching the dense path. The existing non-gated MoE special case (w3 scaling kept at 1.0 to avoid double-scaling) must be preserved.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.