SmolVLM2: RuntimeError: Unsloth: Failed to make input require gradients!

[anderlem]

When loading SmolVLM2 ("HuggingFaceTB/SmolVLM2-2.2B-Instruct") with Unsloth, I get

File "lib/python3.12/site-packages/unsloth_zoo/peft_utils.py", line 216, in requires_grad_pre_hook
raise RuntimeError("Unsloth: Failed to make input require gradients!")
RuntimeError: Unsloth: Failed to make input require gradients!

I think this is because here
https://github.com/huggingface/transformers/blob/8ac2b916b042b1f78b75c9eb941c0f5d2cdd8e10/src/transformers/models/smolvlm/modeling_smolvlm.py#L396-L399
the inputs are not passed as positional arguments, and therefore the requires_grad_pre_hook does not get the inputs (cf. the docs at https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.module.register_module_forward_pre_hook.html)

[anderlem]

When I change the smolvlm modeling class to pass the inputs as positional argument, that error goes away.
But then I get a new error:

  File "lib/python3.12/site-packages/unsloth/models/_utils.py", line 1314, in _unsloth_pre_compute_loss
outputs = self._old_compute_loss(model, inputs, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "lib/python3.12/site-packages/transformers/trainer.py", line 4154, in compute_loss
loss *= self.accelerator.num_processes if self.args.n_gpu <= 1 else self.args.n_gpu
RuntimeError: Output 0 of UnslothFusedLossBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.