Skip to content

Fix LoRA hot swapping #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Fix LoRA hot swapping #116

wants to merge 2 commits into from

Conversation

pbarker
Copy link

@pbarker pbarker commented Apr 17, 2025

LoRA hot swapping is broken as documented in unslothai/unsloth#2322

This is due to an overly permissive regex which when loading multiple adapters which will apply the first adapter correctly but when applying the second adapter it will target layers in the first adapter

The main change is a tighter regex in get_peft_regex

This also formats some of the code to be more up to spec with ruff linting.

I tested this with qwen-vl-2.5 3b and everything appears to work

@danielhanchen
Copy link
Contributor

Thanks for the PR - sorry on the delay! @rolandtannous or @mmathew23 could you guys check if this logic works as expected - appreciate it :)

@rolandtannous
Copy link
Collaborator

rolandtannous commented May 4, 2025

@pbarker @danielhanchen

Test results

Just finished testing this and this fix failed for me.
Jupyterlab on private VM with A100 GPU, with latest unsloth, unsloth_zoo with this fix manually applied to peft_utils.py.
The reason why I think it fails, is that load_adapter method, is a peft library method and not an unsloth method and

get_peft_regex()

is not called when applying the

load_adapter()

method

when is get_peft_regex() actually called?

get_peft_regex() is called when applying the get_peft_model for FastVisionModel or FastLanguageModel

In fact, If you look at the Error trace strack in the testing notebook i've attached, you'll see that the call to load_adapter calls the peft library immediately and there is no intermediary call to get_peft_regex()

Test Notebook

Here is the E2E testing notebook I used based on the code shared by @pbarker here :
test_notebook

Existing peft issue

This seems to be an issue related to the peft library and has been discussed a few times in the peft repo github issues section. Examples:
huggingface/peft#957
huggingface/peft#2388

Workaround

There is a workaround for swapping lora adapters using the hotswap_adapter method. huggingface hotswap method.

The code would look something like this

from peft.utils.hotswap import hotswap_adapter
model_id = "unsloth/Qwen2.5-VL-7B-Instruct"

base_model, model_processor = FastVisionModel.from_pretrained(
    model_id,
    load_in_4bit=False,
)
FastVisionModel.for_inference(base_model)

model = FastVisionModel.get_peft_model(
    base_model,
    finetune_vision_layers=True,  # False if not finetuning vision layers
    finetune_language_layers=True,  # False if not finetuning language layers
    finetune_attention_modules=True,  # False if not finetuning attention layers
    finetune_mlp_modules=True,  # False if not finetuning MLP layers
    r=8,  # The larger, the higher the accuracy, but might overfit
    lora_alpha=16,  # Recommended alpha == r at least
    lora_dropout=0.1,
    bias="none",
    random_state=3407,
    use_rslora=False,  # We support rank stabilized LoRA
    loftq_config=None,  # And LoftQ
    use_fast=True,
    # target_modules = "all-linear", # Optional now! Can specify a list if needed
)

hotswap_adapter(model, "./outputs/checkpoint-15", adapter_name="default")
hotswap_adapter(model, "./outputs/checkpoint-30", adapter_name="default")

Here is an End to End notebook that illustrates the use of hotswap_adapter :
workaround_notebook

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants