[Feature] FastVisionModel

[Vinayyyy7]

I Get A Lot Of Errors While Trying To FineTune Any Model Using FastModel Or FastVisionModel Instead Of FastLanguageModel While Using MultiGPU device_map="auto/balanced"

Request To Add MultiGPU Support For FastVisionModel And FastModel Just Like Language One.

Error :

The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 151645, 'bos_token_id': None}.
==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 2
   \\   /|    Num examples = 262,751 | Num Epochs = 1 | Total steps = 32,844
O^O/ \_/ \    Batch size per device = 2 | Gradient accumulation steps = 4
\        /    Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8
 "-____-"     Trainable parameters = 34,865,152 of 2,162,397,184 (1.61% trained)
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
NotImplementedError: Cannot access storage of TensorWrapper

The above exception was the direct cause of the following exception:

Unsupported                               Traceback (most recent call last)
/tmp/ipykernel_1322/773422404.py in <cell line: 0>()
----> 1 trainer_stats = trainer.train()

/kaggle/working/unsloth_compiled_cache/UnslothSFTTrainer.py in wrapper(self, *args, **kwargs)
     51         if hasattr(self, 'model') and hasattr(self.model, "for_training"):
     52             self.model.for_training()
---> 53         output = f(self, *args, **kwargs)
     54         # Return inference mode
     55         if hasattr(self, 'model') and hasattr(self.model, "for_inference"):

/usr/local/lib/python3.11/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   2323                 hf_hub_utils.enable_progress_bars()
   2324         else:
-> 2325             return inner_training_loop(
   2326                 args=args,
   2327                 resume_from_checkpoint=resume_from_checkpoint,

/usr/local/lib/python3.11/dist-packages/unsloth_zoo/compiler.py in _fast_inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)

/kaggle/working/unsloth_compiled_cache/UnslothSFTTrainer.py in training_step(self, *args, **kwargs)
   1023     def training_step(self, *args, **kwargs):
   1024         with self.maybe_activation_offload_context:
-> 1025             return super().training_step(*args, **kwargs)
   1026 
   1027     def log(self, logs: dict[str, float], start_time: Optional[float] = None) -> None:

/usr/local/lib/python3.11/dist-packages/unsloth/models/_utils.py in _unsloth_training_step(self, model, inputs, num_items_in_batch)

/kaggle/working/unsloth_compiled_cache/UnslothSFTTrainer.py in compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
   1012 
   1013     def compute_loss(self, model, inputs, return_outputs = False, num_items_in_batch = None):
-> 1014         outputs = super().compute_loss(
   1015             model,
   1016             inputs,

/usr/local/lib/python3.11/dist-packages/unsloth/models/_utils.py in _unsloth_pre_compute_loss(self, model, inputs, *args, **kwargs)
   1387         )
   1388     pass
-> 1389     outputs = self._old_compute_loss(model, inputs, *args, **kwargs)
   1390     return outputs
   1391 pass

/usr/local/lib/python3.11/dist-packages/unsloth/models/_utils.py in compute_loss(self, model, inputs, return_outputs, num_items_in_batch)

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1773             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1774         else:
-> 1775             return self._call_impl(*args, **kwargs)
   1776 
   1777     # torchrec tests the code consistency with the following code

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1784                 or _global_backward_pre_hooks or _global_backward_hooks
   1785                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1786             return forward_call(*args, **kwargs)
   1787 
   1788         result = None

/usr/local/lib/python3.11/dist-packages/accelerate/utils/operations.py in forward(*args, **kwargs)
    816 
    817     def forward(*args, **kwargs):
--> 818         return model_forward(*args, **kwargs)
    819 
    820     # To act like a decorator so that it can be popped when doing `extract_model_from_parallel`

/usr/local/lib/python3.11/dist-packages/accelerate/utils/operations.py in __call__(self, *args, **kwargs)
    804 
    805     def __call__(self, *args, **kwargs):
--> 806         return convert_to_fp32(self.model_forward(*args, **kwargs))
    807 
    808     def __getstate__(self):

/usr/local/lib/python3.11/dist-packages/torch/amp/autocast_mode.py in decorate_autocast(*args, **kwargs)
     42     def decorate_autocast(*args, **kwargs):
     43         with autocast_instance:
---> 44             return func(*args, **kwargs)
     45 
     46     decorate_autocast.__script_unsupported = (  # type: ignore[attr-defined]

/usr/local/lib/python3.11/dist-packages/peft/peft_model.py in forward(self, input_ids, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, task_ids, **kwargs)
   1848             with self._enable_peft_forward_hooks(**kwargs):
   1849                 kwargs = {k: v for k, v in kwargs.items() if k not in self.special_peft_forward_args}
-> 1850                 return self.base_model(
   1851                     input_ids=input_ids,
   1852                     attention_mask=attention_mask,

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)
   1773             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1774         else:
-> 1775             return self._call_impl(*args, **kwargs)
   1776 
   1777     # torchrec tests the code consistency with the following code

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)
   1879 
   1880         try:
-> 1881             return inner()
   1882         except Exception:
   1883             # run always called hooks if they have not already been run

/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py in inner()
   1827                 args = bw_hook.setup_input_hook(args)
   1828 
-> 1829             result = forward_call(*args, **kwargs)
   1830             if _global_forward_hooks or self._forward_hooks:
   1831                 for hook_id, hook in (

/usr/local/lib/python3.11/dist-packages/peft/tuners/tuners_utils.py in forward(self, *args, **kwargs)
    220 
    221     def forward(self, *args: Any, **kwargs: Any):
--> 222         return self.model.forward(*args, **kwargs)
    223 
    224     def _pre_injection_hook(self, model: nn.Module, config: PeftConfig, adapter_name: str) -> None:

/usr/local/lib/python3.11/dist-packages/accelerate/hooks.py in new_forward(module, *args, **kwargs)
    173                 output = module._old_forward(*args, **kwargs)
    174         else:
--> 175             output = module._old_forward(*args, **kwargs)
    176         return module._hf_hook.post_forward(module, output)
    177 

/kaggle/working/unsloth_compiled_cache/unsloth_compiled_module_qwen3_vl.py in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, cache_position, logits_to_keep, **kwargs)
   1209         **kwargs: Unpack[TransformersKwargs],
   1210     ) -> Union[tuple, Qwen3VLCausalLMOutputWithPast]:
-> 1211         return Qwen3VLForConditionalGeneration_forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, cache_position, logits_to_keep, **kwargs)
   1212 
   1213     def prepare_inputs_for_generation(

/usr/local/lib/python3.11/dist-packages/torch/_dynamo/external_utils.py in nonrecursive_disable_wrapper(*args, **kwargs)
    194     @functools.wraps(fn)
    195     def nonrecursive_disable_wrapper(*args: _P.args, **kwargs: _P.kwargs) -> _R:
--> 196         return fn(*args, **kwargs)
    197 
    198     return nonrecursive_disable_wrapper

/kaggle/working/unsloth_compiled_cache/unsloth_compiled_module_qwen3_vl.py in Qwen3VLForConditionalGeneration_forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, cache_position, logits_to_keep, **kwargs)
   1114         torch._dynamo.mark_dynamic(_hidden_states, 1)
   1115         torch._dynamo.mark_dynamic(labels, 1)
-> 1116         loss = unsloth_fused_ce_loss(
   1117             trainer              = None,
   1118             hidden_states        = _hidden_states,

/usr/local/lib/python3.11/dist-packages/unsloth_zoo/fused_losses/cross_entropy_loss.py in unsloth_fused_ce_loss(trainer, hidden_states, lm_head_weight, lm_head_bias, labels, mask, n_items, scaling, target_gb, torch_compile, overwrite, **kwargs)
    362     scaling = scaler.get_scale() if scaler is not None else scaling
    363     if hasattr(scaling, "get_scale"): scaling = scaling.get_scale()
--> 364     return apply_autograd_function(UnslothFusedLoss, dict(
    365         loss_function = compute_fused_ce_loss,
    366         hidden_states = hidden_states,

/usr/local/lib/python3.11/dist-packages/unsloth_zoo/fused_losses/cross_entropy_loss.py in apply_autograd_function(autograd, mapping)
     39 def apply_autograd_function(autograd, mapping):
     40     parameters, defaults = _get_mapping(autograd)
---> 41     return getattr(autograd, "apply")(*(
     42         mapping.get(old_key, default) \
     43         for old_key, default in zip(parameters, defaults)

/usr/local/lib/python3.11/dist-packages/torch/autograd/function.py in apply(cls, *args, **kwargs)
    579             # See NOTE: [functorch vjp and autograd interaction]
    580             args = _functorch.utils.unwrap_dead_wrappers(args)
--> 581             return super().apply(*args, **kwargs)  # type: ignore[misc]
    582 
    583         if not is_setup_ctx_defined:

/usr/local/lib/python3.11/dist-packages/unsloth_zoo/fused_losses/cross_entropy_loss.py in forward(ctx, loss_function, hidden_states, lm_head_weight, lm_head_bias, labels, mask, n_items, scaling, shift_labels, target_gb, torch_compile, overwrite, extra_kwargs)
    302         for (grad_inputs_j, hidden_states_j, labels_j,) in \
    303             zip(__grad_inputs, __shift_states, __shift_labels,):
--> 304             accumulate_chunk(
    305                 n_chunks = n_chunks,
    306                 grad_inputs_j = grad_inputs_j,

/usr/local/lib/python3.11/dist-packages/torch/_dynamo/eval_frame.py in compile_wrapper(*args, **kwargs)
    839                         cur_exn.__cause__.with_traceback(None)
    840                         cur_exn = cur_exn.__cause__
--> 841                     raise e.with_traceback(None) from e.__cause__  # User compiler error
    842                 except ShortenTraceback as e:
    843                     # Failures in the backend likely don't have useful

Unsupported: NotImplementedError/UnsupportedFakeTensorException when running FX node
  Explanation: Dynamo failed to run FX node with fake tensors: call_function <function _autograd_grad at 0x7adc2d2d8180>(*((GradTrackingTensor(lvl=1, value=
        FakeTensor(..., device='cuda:0', size=())
    ),), [GradTrackingTensor(lvl=1, value=
        FakeTensor(..., device='cuda:1', size=(s97, 2048), dtype=torch.float16,
                   requires_grad=True)
    )]), **{'create_graph': True}): got NotImplementedError('Cannot access storage of TensorWrapper')
  Hint: If the op is a PyTorch op, please file an issue to PyTorch.

  Developer debug context: 

 For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0087.html

from user code:
   File "/usr/local/lib/python3.11/dist-packages/unsloth_zoo/fused_losses/cross_entropy_loss.py", line 276, in accumulate_chunk
    (chunk_loss, (unscaled_loss,)) = torch.func.grad_and_value(
  File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/apis.py", line 449, in wrapper
    return eager_transforms.grad_and_value_impl(
  File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/vmap.py", line 47, in fn
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.11/dist-packages/torch/_functorch/eager_transforms.py", line 1390, in grad_and_value_impl
    flat_grad_input = _autograd_grad(

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

I've Tried Setting LoRA Very Low Yet It Doesn't Work (On MultiGPU) So Not Just An OutOfMemory Error. And To Work On Single GPU LoRA Needs To Be Very Low Which Is Really Not Worth Tuning.

[urbanspr1nter]

I train (SFT) multi-GPU with FastVisionModel.
How are you invoking the training script? Are you using accelerate?
device_map should be something like: {"": torch.cuda.current_device()}

[Vinayyyy7]

I train (SFT) multi-GPU with FastVisionModel. How are you invoking the training script? Are you using accelerate? device_map should be something like: {"": torch.cuda.current_device()}

Nah Not Using accelerate should I? Also I Would Try The device map Config You Told.

It's Like The Model Is Distributed On Both GPUs Properly (Not So Equally Btw) But On Training Start In Crashes.

Thanks I Would Try Your Config Suggestion

[urbanspr1nter]

Yes, per the docs: https://docs.unsloth.ai/basics/multi-gpu-training-with-unsloth

You should use accelerate. Run accelerate config first to configure your environment. I do it this way and works reliably.

[Vinayyyy7]

tried using device_map={"": torch.cuda.current_device()} It loads the model only on a single gpu and other is not utilized.

accelerate default_config.yaml file is :

{
  "compute_environment": "LOCAL_MACHINE",
  "debug": false,
  "distributed_type": "MULTI_GPU",
  "downcast_bf16": false,
  "enable_cpu_affinity": false,
  "machine_rank": 0,
  "main_training_function": "main",
  "mixed_precision": "no",
  "num_machines": 1,
  "num_processes": 2,
  "rdzv_backend": "static",
  "same_network": false,
  "tpu_use_cluster": false,
  "tpu_use_sudo": false,
  "use_cpu": false
}

MY FULL TRAINING SCRIPT IS LIKE :

from unsloth import FastVisionModel # FastLanguageModel for LLMs
import torch

model, tokenizer = FastVisionModel.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct", # OR "Qwen/Qwen3-VL-2B-Thinking" (Try Others Like InternVL)
    load_in_4bit = False, 
    device_map = {"": torch.cuda.current_device()},
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context
)

model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = False, # False if not finetuning vision layers
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = 786,           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 786,  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
    # target_modules = "all-linear", # Optional now! Can specify a list if needed
)


from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from transformers import DataCollatorForSeq2Seq, AutoTokenizer

# Load tokenizer
base_tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct",
    trust_remote_code=True
)

if base_tokenizer.pad_token is None:
    base_tokenizer.pad_token = base_tokenizer.eos_token

# Load JSONL directly
dataset = load_dataset("json", data_files="/kaggle/input/my-dataset/DataVL.jsonl")
train_dataset = dataset["train"]

print(f"Loaded {len(train_dataset)} samples")
print(f"Sample:\n{train_dataset[0]}")

# Preprocessing function
def format_messages(examples):
    """Format messages as text"""
    formatted_texts = []
    
    for messages in examples['messages']:
        text = ""
        for msg in messages:
            role = msg.get('role', '').upper()
            content = msg.get('content', [])
            
            msg_text = ""
            for item in content:
                if isinstance(item, dict) and item.get('type') == 'text':
                    msg_text += item.get('text', '')
            
            text += f"{role}: {msg_text}\n"
        
        formatted_texts.append(text.strip())
    
    tokenized = base_tokenizer(
        formatted_texts,
        truncation=True,
        max_length=2048,
        padding="max_length",
        return_tensors=None
    )
    
    tokenized["labels"] = [ids[:] for ids in tokenized["input_ids"]]
    
    return tokenized

# Apply preprocessing
train_dataset = train_dataset.map(
    format_messages,
    batched=True,
    batch_size=16,
    remove_columns=['messages'],
    desc="Formatting and tokenizing"
)

print(f"Preprocessed dataset ready!")

FastVisionModel.for_training(model)

trainer = SFTTrainer(
    model=model,
    tokenizer=base_tokenizer,
    data_collator=DataCollatorForSeq2Seq(base_tokenizer, pad_to_multiple_of=8),
    train_dataset=train_dataset,
    args=SFTConfig(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        ddp_find_unused_parameters = False, # Acc To unsloth docs
        # max_steps=30,
        num_train_epochs=1,
        learning_rate=2e-5,
        logging_steps=1, # For Each Step
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        save_total_limit=2, # To Save Only 2 Latest Checkpoints
        seed=3407,
        output_dir="/kaggle/working/checkpoints",
        report_to="none",
        remove_unused_columns=False,
        max_length=2048,
    ),
)

trainer_stats = trainer.train()

Error Got When Using device_map={"": torch.cuda.current_device()}

[rank1]: ValueError: You can't train a model that has been loaded in 8-bit or 4-bit precision on a different device than the one you're training on. Make sure you loaded the model on the correct device using for example `device_map={'':torch.cuda.current_device()}` or `device_map={'':torch.xpu.current_device()}`

Error Got When Using device_map="balanced" (Acc To Unsloth Docs)

[rank1]: ValueError: You can't train a model that has been loaded with `device_map='auto'` in any distributed mode. Please rerun your script specifying `--num_processes=1` or by launching with `python {{myscript.py}}`.

Also I have tried both

accelerate launch train.py

&

torchrun --nproc_per_node N_GPUS -m train.py

Many Issues Online Suggests That Only Use One GPU (If It Has Enough Higer VRAM) And Yes It Works I'have tested With InternVL 3.5 1B On A Single GPU That Training Works Well Using Same Code Above.

@urbanspr1nter Thanks.

[urbanspr1nter]

Depending on how much VRAM you have, do consider that per_device_train_batch_size=2 means that you'll have a larger batch size per GPU since the batch size is computed by N_GPUs x per device train batch. Maybe try to reduce batch size to 1 first?

[sirenhagger]

i meet the same issue when multi-gpu training using accelerate:

error message like:
[rank1]: ValueError: You can't train a model that has been loaded in 8-bit or 4-bit precision on a different device than the one you're training on. Make sure you loaded the model on the correct device using for example device_map={'':torch.cuda.current_device()} or `device_map={'':torch.xpu.current_device()}

when using single gpu or num_prcocess=1 under accelerate config, training is successful.

no matter i use {'':torch.cuda.current_device()} or "balanced" (i bypass the auto problem under accelerate), they all meet the same error message, it seems 4bit model is not allowed for multi-gpus training.

can this problem be solved?

[Vinayyyy7]

@sirenhagger .

based on reports so far (including mine and yours), the issue consistently reproduces when using multi-GPU training wih accelerate on (4or 8bit) models (and 16bit which I used), while single-GPU works as expected. Using different device_map options (auto, balanced, or forcing {'': torch.cuda.current_device()}) does not appear to resolve the error.

it’s unclear whether this is a limitation of Unsloth’s current FastVisionModel or accelerate & unsloth coordination ... I think issue must be with unsloth not accelerate.

[Vinayyyy7]

@sirenhagger got a update, I don't know which model you tried maybe a normal text-generation model with FastLanguageModel class or a multimodal VLM using FastVisionModel class. But I did some fixes since I had to come back to this issue after a project of mine now requires this to be fixed.

After the fixes I made, I tested on kaggle with 2x T4 GPUs on Kaggle ENV, on Qwen/Qwen3-VL-2B-Instruct with device_map="auto"

IT WORKED & NO CRASH ERRORS...

both gpus were utilized.

==((====))==  Unsloth - 2x faster free finetuning | Num GPUs used = 2
   \\   /|    Num examples = 10,000 | Num Epochs = 1 | Total steps = 1,250
O^O/ \_/ \    Batch size per device = 2 | Gradient accumulation steps = 4
\        /    Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8
 "-____-"     Trainable parameters = 139,460,608 of 2,266,992,640 (6.15% trained)
Step | Training Loss
-- | --
1 | 46.795200
2 | 109.610000
3 | 40.122600
4 | 44.093400
5 | 32.455600
6 | 47.650400
7 | 60.632800
8 | 24.140800
9 | 63.971800
10 | 19.096700
11 | 20.428400
12 | 17.989900
13 | 10.218600
14 | 20.911900
15 | 16.450800
16 | 6.303300
17 | 4.037200
18 | 4.878000
19 | 2.447200
20 | 2.050200

HERE ARE BUILT FILES YOU CAN USE IF YOU LIKE TO. FOR UNSLOTH & UNSLOTH-ZOO IT REQUIRED FIXING BOTH LIBRARIES...

unsloth-2025.12.9.tar.gz

unsloth_zoo-2025.12.7.tar.gz

Also Made PRs expecting them to be merged in offical repo soon...
#3809
unslothai/unsloth-zoo#423

[sirenhagger]

@Vinayyyy7 Thank you for your sharing. I did some test based on your updated unsloth and unsloth_zoo. The following are details:

1. unsloth model selection

model: unsloth/Qwen3-VL-4B-Instruct-unsloth-bnb-4bit

I select the specific version under unsloth and it is a 4bit version. One card post training only costs 5.5GIB GPU memory, my input token length is <1500. This is much more efficient than bnb_qlora + trl under bf16 with original qwen/Qwen3-VL-4B-Instruct (Such setting costs 18GIB).

My project will use 32B model, thus why i prefer to use 4bit version of unsloth as the base model.

2. Accelerate 1p training

Accelerate config:
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
use_cpu: false

Training trail:
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\ /| Num examples = 8 | Num Epochs = 20 | Total steps = 160
O^O/ _/ \ Batch size per device = 1 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 1 | Total batch size (1 x 1 x 1) = 1
"-____-" Trainable parameters = 25,952,256 of 4,463,768,064 (0.58% trained)
0%| | 0/160 [00:00<?, ?it/s]Unsloth: Will smartly offload gradients to save VRAM!
{'loss': 3.1778, 'grad_norm': 5.62775993347168, 'learning_rate': 5.625e-06, 'epoch': 1.25}
{'loss': 2.8907, 'grad_norm': 3.168405055999756, 'learning_rate': 9.791666666666666e-06, 'epoch': 2.5}
{'loss': 2.7466, 'grad_norm': 2.2995803356170654, 'learning_rate': 9.097222222222223e-06, 'epoch': 3.75}
{'loss': 2.4069, 'grad_norm': 2.1300830841064453, 'learning_rate': 8.402777777777779e-06, 'epoch': 5.0}
{'loss': 2.228, 'grad_norm': 1.7144867181777954, 'learning_rate': 7.708333333333334e-06, 'epoch': 6.25}
{'loss': 2.0032, 'grad_norm': 1.4950071573257446, 'learning_rate': 7.013888888888889e-06, 'epoch': 7.5}
{'loss': 1.8706, 'grad_norm': 1.1102083921432495, 'learning_rate': 6.319444444444445e-06, 'epoch': 8.75}
{'loss': 1.7424, 'grad_norm': 1.3088957071304321, 'learning_rate': 5.625e-06, 'epoch': 10.0}
{'loss': 1.596, 'grad_norm': 1.0487256050109863, 'learning_rate': 4.930555555555556e-06, 'epoch': 11.25}
{'loss': 1.5294, 'grad_norm': 1.584958791732788, 'learning_rate': 4.236111111111111e-06, 'epoch': 12.5}
{'loss': 1.3732, 'grad_norm': 2.037498712539673, 'learning_rate': 3.5416666666666673e-06, 'epoch': 13.75}
{'loss': 1.2874, 'grad_norm': 1.8874232769012451, 'learning_rate': 2.8472222222222224e-06, 'epoch': 15.0}
{'loss': 1.2005, 'grad_norm': 1.8668657541275024, 'learning_rate': 2.152777777777778e-06, 'epoch': 16.25}
{'loss': 1.1333, 'grad_norm': 1.8730494976043701, 'learning_rate': 1.4583333333333335e-06, 'epoch': 17.5}
{'loss': 1.0659, 'grad_norm': 1.9273625612258911, 'learning_rate': 7.63888888888889e-07, 'epoch': 18.75}
{'loss': 1.042, 'grad_norm': 2.1040780544281006, 'learning_rate': 6.944444444444444e-08, 'epoch': 20.0}

The convergence is good, and test is good althrough training dataset has only 8 samples here.

2. Accelerate 2p training

Accelerate config:
num_processes: 1

Training trail:
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\ /| Num examples = 8 | Num Epochs = 20 | Total steps = 80
2P DDP

O^O/ _/ \ Batch size per device = 1 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 2 | Total batch size (1 x 1 x 2) = 2
"--" Trainable parameters = 25,952,256 of 2,921,705,984 (0.89% trained)
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\ /| Num examples = 8 | Num Epochs = 20 | Total steps = 80
O^O/ _/ \ Batch size per device = 1 | Gradient accumulation steps = 1
\ / Data Parallel GPUs = 2 | Total batch size (1 x 1 x 2) = 2
"--" Trainable parameters = 25,952,256 of 2,921,705,984 (0.89% trained)
{'loss': 3.1028, 'grad_norm': 0.0, 'learning_rate': 9.861111111111112e-06, 'epoch': 2.5}
{'loss': 3.1485, 'grad_norm': 0.0, 'learning_rate': 8.472222222222223e-06, 'epoch': 5.0}
{'loss': 3.099, 'grad_norm': 0.0, 'learning_rate': 7.083333333333335e-06, 'epoch': 7.5}
{'loss': 3.152, 'grad_norm': 0.0, 'learning_rate': 5.694444444444445e-06, 'epoch': 10.0}
{'loss': 3.1642, 'grad_norm': 0.0, 'learning_rate': 4.305555555555556e-06, 'epoch': 12.5}
{'loss': 3.0917, 'grad_norm': 0.0, 'learning_rate': 2.916666666666667e-06, 'epoch': 15.0}
{'loss': 3.143, 'grad_norm': 0.0, 'learning_rate': 1.527777777777778e-06, 'epoch': 17.5}
{'loss': 3.1109, 'grad_norm': 0.0, 'learning_rate': 1.3888888888888888e-07, 'epoch': 20.0}

Revision:
unsloth_zoo/compiler.py
-Remove variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}
-No need for alora for now
variant_kwarg_keys = "variant_kwargs = {k: kwargs.pop(k, None) for k in VARIANT_KWARG_KEYS}"
variant_found = source.find(variant_kwarg_keys)
if variant_found != -1:
variant_end = source.find("\n", variant_found + len(variant_kwarg_keys))
source = source.replace(source[variant_found : variant_end], "")

From the update you give, i have to uncomment the above code under unsloth_zoo/compiler.py, otherwise linear_peft_forward.py under compiled file will raise VARIANT_KWARG_KEYS not defined.

As you revise the device_map code under unsloth/models/loader.py, now DDP runs successfully. However, it seems my 4bit version did not converge. I guess there maybe a mismatch between distributed communication dtype and model param dtype.

I may try to locate the problem or just use fp16 version model in the future.

3. Accelerate+Deepspeed(zero2) 2p training

Accelerate config:
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
deepspeed_multinode_launcher: standard
offload_optimizer_device: none
offload_param_device: none
zero3_init_flag: true
zero_stage: 2
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

Error:
File "/home/xt/miniconda3/envs/Pytorch_2x_Unsloth_Qwen3/lib/python3.12/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 848, in independent_gradient_partition_epilogue
[rank0]: self.reduce_ipg_grads()
[rank0]: File "/home/xt/miniconda3/envs/Pytorch_2x_Unsloth_Qwen3/lib/python3.12/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1504, in reduce_ipg_grads
[rank0]: self.average_tensor(bucket.buffer[bucket.index].narrow(0, 0, bucket.elements), comm_dtype)
[rank0]: ~~~~~~~~~~~~~^^^^^^^^^^^^^^
[rank0]: IndexError: list index out of range

Well, it seems during gradient partition, the communication has a problem, i have checked that bucket.buffer ask for bf16 type, but bucket.buffer is empty under unsloth framwork. Qlora+Accelerate(deepspeed zero2) runs successfully based on qwen/Qwen3-VL-4B-Instruct model.

4. Accelerate 2p GSPO/GRPO Training

same with 2. Accelerate 2p training
Accelerate config:
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 2
rdzv_backend: static
same_network: true
use_cpu: false

Error:
"/home/xt/trainer/Qwen3VL_instruct/unsloth_compiled_cache/UnslothGRPOTrainer.py", line 3001, in compute_loss
[rank1]: logit_softcapping = getattr(model.config, "final_logit_softcapping", 0) # Gemma
[rank1]: ^^^^^^^^^^^^
[rank1]: File "/home/xt/miniconda3/envs/Pytorch_2x_Unsloth_Qwen3/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1962, in getattr
[rank1]: raise AttributeError(
[rank1]: AttributeError: 'DistributedDataParallel' object has no attribute 'config'

Just alternate the trl sft trainer with grpo trainer with additional reward functions. But i seems the compute_loss function under GRPOunslothTrainer did not consider distribute condition.

Revision:
unsloth/modele/rl_replacement.py
near line 720, add -
-Branch for DDP warpper
if torch.distributed.is_initialized():
model = model.module

Training trail:
'loss': 0.0, 'grad_norm': 1.6764026440796442e-05, 'learning_rate': 1.8750000000000003e-06, 'num_tokens': 5429.0, 'completions/mean_length': 74.0, 'completions/min_length': 74.0, 'completions/max_length': 74.0, 'completions/clipped_ratio': 0.0, 'completions/mean_terminated_length': 74.0, 'completions/min_terminated_length': 74.0, 'completions/max_terminated_length': 74.0, 'rewards/formatting_reward_func/mean': 1.6666666269302368, 'rewards/formatting_reward_func/std': 0.0, 'rewards/correctness_reward_func/mean': 0.0, 'rewards/correctness_reward_func/std': 0.0, 'reward': 1.6666666269302368, 'reward_std': 0.0, 'frac_reward_zero_std': 1.0, 'completion_length': 74.0, 'kl': 0.0015514789847657084, 'epoch': 0.5}
5%|█████ | 4/80 [01:28<23:38, 18.67s/it]

It ran successfully. But, whether reward signal can converge or not is another problem. I think it also suffers from the convergence problem like sft trainer.

Thank you for your sharing again, i do not know whether i can process the commuication dtype problem (or even this is not the cause) in my scenario. But, i try to use unsloth as possible.

[Vinayyyy7]

Hi @sirenhagger , thank you again for the incredibly detailed testing...

just pushed some updates to the PRs to incorporate your findings and harden the implementation:

1. Fixes

• GRPO Multi-GPU Fix (unsloth/models/rl_replacements.py): I added the if torch.distributed.is_initialized(): model = model.module check you identified. this should resolve the error crash for GRPO.:

AttributeError: 'DistributedDataParallel' ... has no attribute 'config'

• DDP Safety Enforced (loader.py & vision.py): changed the warning to a strict ValueError. If a user tries device_map="auto/balanced" in a distributed env (that causes the model splitting/gradient mismatch crash), it will now explicitly error out and instruct them to use device_map=None for Data Parallelism. (Multi-GPU fix)

---

2. Regarding DDP Convergence

the poor convergence loss you saw in DDP mode:

Since this fix forces standard Data Parallelism (splitting the batch, not the model), each GPU trains on its own subset of data.

as mentioned, test dataset was small (like 8 examples). In a 2-GPU DDP setup, that effectively means each GPU only got 4 examples.
this can be noisy for a small training steps, and atleast for 15-20 steps loss normally fluctuates around same number then only it goes a little down...

Suggestion: you can try running the DDP test with a larger dataset, maybe 100-200 examples to see if the loss stabilizes over time?

If it still fails to converge on a larger run, we can treat that as a separate research issue for 4-bit DDP Gradient Precision, but the crash itself should now be resolved.