[LoRA] Add an end-to-end single-adapter runtime path for Qwen2

[MagellaX]

What this does

This adds a correctness-first runtime LoRA path for Qwen2/Qwen2.5. Instead of merging the adapter into the base weights, convert_weight --lora-mode runtime packages the PEFT A/B tensors separately and compile --lora-adapter adds the low-rank branches to the model graph. The existing PEFT merge flow remains the default.

This is intended to establish the runnable end-to-end path requested in apache/tvm#18890 before we settle the longer-term pool/runtime API tracked in mlc-llm#3446.

Current scope

• Qwen2/Qwen2.5, q0f16, TP=1
• Standard LoRA and rsLoRA
• One fixed adapter per compiled artifact
• No request-time adapter switching, mixed-adapter batching, LoRAPool, or Punica kernels yet

Commands

python -m mlc_llm convert_weight "$BASE_MODEL" \
  --source "$BASE_MODEL" \
  --source-format huggingface-safetensor \
  --quantization q0f16 \
  --device cuda \
  --lora-adapter "$LORA_ADAPTER" \
  --lora-mode runtime \
  -o dist/runtime-lora

python -m mlc_llm compile "$BASE_MODEL" \
  --quantization q0f16 \
  --device cuda \
  --lora-adapter "$LORA_ADAPTER" \
  -o dist/runtime-lora.so

python -m mlc_llm chat dist/runtime-lora \
  --model-lib dist/runtime-lora.so \
  --device cuda

Validation

Tested on an A100 with CUDA 12.8 using Qwen/Qwen2.5-0.5B-Instruct and vindows/qwen2.5-0.5b-text-to-sql:

• Runtime-separate and PEFT-merged paths produced token-for-token identical output on 15/15 prompts
• Adapter parameters add 16.78 MiB; the current unoptimized runtime path adds about 226-228 MiB at the process level
• Full unit selection: 52 passed, 13 skipped, 315 deselected
• Ruff check and format checks passed for all changed files

I’m opening this as a draft so we can validate the end-to-end boundary and agree whether the next patch should introduce a standalone adapter/LoRAPool runtime object before adding request-time adapter IDs and multi-adapter batching.

[gemini-code-assist]

Code Review

This pull request introduces support for compiling and packaging separate low-rank PEFT LoRA adapter tensors into the runtime graph (runtime LoRA) for Qwen2 models using q0f16 quantization and single-GPU execution. It adds CLI options for runtime LoRA mode, implements PEFT adapter configuration validation, and integrates low-rank linear layers into the Qwen2 attention and MLP blocks. A critical issue was identified in qwen2_model.py where op.concat incorrectly uses the dim parameter instead of axis, which will cause a compilation failure in TVM Relax.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[MagellaX]

@babusid @MasterJH5574 any thoughts guys??