👁️ [GRPO] Add VLM training capabilities to the trainer - #3072

Merged

qgallouedec merged 122 commits into

huggingface:mainfrom

CompN3rd:grpo_vlm

Jul 23, 2025

Commits on Mar 13, 2025

[GRPO] vlm training
Marc Kassubeck
committed
6ca8bde
test vlm training
Marc Kassubeck
committed
8f34987

Commits on Mar 20, 2025

Merge remote-tracking branch 'upstream/main' into grpo_vlm
Marc Kassubeck
committed
328ef46

Commits on Mar 25, 2025

Merge remote-tracking branch 'upstream/main' into grpo_vlm
Marc Kassubeck
committed
e94b5fa

Commits on Jul 9, 2025

Merge branch 'main' into grpo_vlm
kashif
authored
eb453d9
fix merge
kashif
authored
9f6517d
fix imports
kashif
authored
ac9ba6f
fix merge issues and some tests
kashif
authored
3b38035
fix config merge
kashif
authored
3694f35
fix tests
kashif
authored
c79c97d
undo change from merge
kashif
authored
ef89aec
fix merge typo
kashif
authored
1e7111a
test smolvm
kashif
authored
4f2c19d

Commits on Jul 10, 2025

do not set to false with vlm
kashif
committed
1923a5a
enable vllm
kashif
committed
01fa042
add docs
kashif
committed
f9521ae
validate images helper
kashif
committed
e552d13
Update docs/source/grpo_trainer.md

kashif
and
sergiopaniego
authored
7eddba9
set the _set_signature_columns_if_needed
kashif
committed
c965d0e
make tests a bit more light weight
kashif
committed
ab80abc
Added grpo_vlm.py example script
sergiopaniego
committed
88e0b77
Updated script example
sergiopaniego
committed
cecdaec

Commits on Jul 11, 2025

Merge branch 'main' into grpo_vlm
kashif
authored
7d0d001
formatting
kashif
committed
41d1e11
log probs now use visual context
kashif
committed
a34b5f7
remove duplicate import
kashif
committed
a619898
link example script
kashif
committed
1c02bb9
use AutoProcessor for vision_models
kashif
committed
5b3eaf7
split_tensor_dict and split_tensor_dict handle BatchFeature objects
kashif
committed
215e6fb
Only shuffle/split tensors that have the expected batch dimension
kashif
committed
99da8e8
Script more general
sergiopaniego
committed
3eb777a
Merge branch 'grpo_vlm' of github.com:CompN3rd/trl into grpo_vlm
sergiopaniego
committed
ed17dff
Merge branch 'main' of github.com:huggingface/trl into grpo_vlm
sergiopaniego
committed
f45f63b
fix _get_per_token_logps_and_entropies
kashif
committed
e6c983b
formatting
kashif
committed
eb9e156
add _prepare_inputs_vlm_mode and move slow tests to test_grpo_slow.py
kashif
committed
a1507fc
add back the tests
kashif
committed
0aba452

Commits on Jul 12, 2025

formatting
kashif
committed
a7b35ef
Merge branch 'main' into grpo_vlm
kashif
authored
bd65822
formatting
kashif
committed
4c2bac5
fix tests
kashif
committed
ce3a7ac
Update docs/source/grpo_trainer.md

kashif
and
qgallouedec
authored
fe9d5ee
Update docs/source/grpo_trainer.md

kashif
and
qgallouedec
authored
e2c17ca
use isinstance BatchFeature
kashif
committed
621d193
removed unused code
kashif
committed
5f49c97
Update trl/trainer/grpo_trainer.py

kashif
and
qgallouedec
authored
a82e432
make get_from_processor_or_tokenizer a function
kashif
committed
6c3d94b
add a setter
kashif
committed
f0355d1
remove unneeded test
kashif
committed
f48d0f6
assume normal configurations
kashif
committed
6b0be5c
review
kashif
committed
ac8c6c4

Commits on Jul 14, 2025

fix test
kashif
committed
81cb565
skip test if no vllm
kashif
committed
cfd0844
integrate fixes from #3725
kashif
committed
3a9f16a
fix log sizes
kashif
committed
f57b98a
Update tests/test_grpo_trainer.py
kashif
authored
c5d0a1a

Commits on Jul 15, 2025

undo changes in tests and move vllm test to slow
kashif
committed
5e791c7
validate_and_preprocess_images to utils
kashif
committed
56dc897
Updated docs and example
sergiopaniego
committed
4729cdc
Update tests/slow/test_grpo_slow.py

kashif
and
sergiopaniego
authored
873821a
Update tests/slow/test_grpo_slow.py

kashif
and
sergiopaniego
authored
784c265
Update tests/slow/test_grpo_slow.py

kashif
and
sergiopaniego
authored
368b751
fix import
kashif
committed
454a4d6
Update import
sergiopaniego
committed
a8e516f
fix repeat_count
kashif
committed
5e3953d
fake_generate takes input_ids
kashif
committed
3c9091e
Merge branch 'main' into grpo_vlm
kashif
authored
4102d0f
formatting
kashif
committed
64dfa81

Commits on Jul 16, 2025

Merge branch 'main' into grpo_vlm
qgallouedec
authored
fa2d349
tiny qwen vl
qgallouedec
committed
4a98b10
Merge branch 'main' into grpo_vlm
kashif
authored
348adf3

Commits on Jul 19, 2025

Merge branch 'main' into grpo_vlm
qgallouedec
authored
aac9f44
zen image initial version
qgallouedec
committed
408ed3f
generate tiny
qgallouedec
committed
8db1520
fix get_from_processor_or_tokenizer, dtype and model loading
qgallouedec
committed
5a576c8
revert unrelated change
qgallouedec
committed
5807aa5
fix tests
kashif
committed
73eddf1
Merge branch 'main' into grpo_vlm
qgallouedec
authored
5b6fa17
Merge branch 'main' into grpo_vlm
qgallouedec
authored
2b79b4b

Commits on Jul 20, 2025

finally it works!
qgallouedec
committed
1a656f7
doc fix
qgallouedec
committed
1c9093c
clean import
qgallouedec
committed
daf4b32
generate tiny models
qgallouedec
committed
ab0ff92
let's go
qgallouedec
committed
994fd91
Update grpo_trainer.py

qgallouedec
and
kashif
authored
07dbb5c
vlm peft
qgallouedec
committed
83e9aae
solved
qgallouedec
committed
d31022f
fix breaking test (inc. slow)
kashif
committed
f43998c
Merge branch 'main' into grpo_vlm
kashif
authored
a8191ae
test qwenv2, IS and liger
qgallouedec
committed
d677874
liger
qgallouedec
committed
d0d9044
fix generate tiny
qgallouedec
committed
95357d2
clean test
qgallouedec
committed
5976ad2
fix comment
qgallouedec
committed
f14401a
fix doc
qgallouedec
committed
e08d342
fix doc
qgallouedec
committed
20f4d14
add test for training with VLM and prompt truncation
qgallouedec
committed
dc66e72
torchvision
qgallouedec
committed
48ae448
add logic to truncate prompt ids if max_prompt_length is set
kashif
committed
f4733ac

Commits on Jul 21, 2025

vllm server support images!
qgallouedec
committed
7f9c835
fix vision import
qgallouedec
committed
b122161
vllm image support
qgallouedec
committed
00d3496
clean truncation
qgallouedec
committed
78f4ecb
log images
qgallouedec
committed
b754d61
clean
qgallouedec
committed
60c14b0

Commits on Jul 22, 2025

Add support for Gemma3 model and update GRPOTrainer initialization
qgallouedec
committed
13e8ac1
style
qgallouedec
committed
fe91966
doc simplification
qgallouedec
committed
6378887
simplify example script
qgallouedec
committed
1935a02
Enhance GRPO training script with additional parameters and improve pixel value handling
qgallouedec
committed
1fbf33a
Merge branch 'main' into grpo_vlm
qgallouedec
authored
488eb16
Update trl/scripts/vllm_serve.py
kashif
authored
f5f164b
Merge branch 'main' into grpo_vlm
kashif
authored
1423e29
Update trl/scripts/vllm_serve.py
kashif
authored
d8460ea
Update trl/trainer/grpo_trainer.py

qgallouedec
and
sergiopaniego
authored
1b757c1
Add tests for split and unsplit pixel values by grid functionality
qgallouedec
committed
b78af71
Merge branch 'grpo_vlm' of https://github.com/compn3rd/trl into grpo_vlm
qgallouedec
committed
578ca66
add a comment
qgallouedec
committed
94f5be5
require vision for some tests
qgallouedec
committed
782f99e
Update tests/test_grpo_trainer.py

qgallouedec
and
sergiopaniego
authored
b264b29
fix require vision
qgallouedec
committed
5b69239
Merge branch 'grpo_vlm' of https://github.com/compn3rd/trl into grpo_vlm
qgallouedec
committed
52a1f29