Online Training - #1341

Draft

jacklanchantin wants to merge 120 commits into

mainfrom

online_training

Commits on Feb 22, 2025

online training components wip
Ilia Kulikov
committed
4c9fb9e

Commits on Feb 23, 2025

wip
Ilia Kulikov
committed
6995cb9
Merge branch 'main' into online_training
Ilia Kulikov
committed
1a4f79a
wip
Ilia Kulikov
committed
8ca6d71
Merge branch 'main' into online_training
Ilia Kulikov
committed
d776eb5

Commits on Feb 28, 2025

wip changes
Ilia Kulikov
committed
d9f3bf0
wip
Ilia Kulikov
committed
bcf05f4

Commits on Mar 5, 2025

more
Ilia Kulikov
committed
e915733

Commits on Mar 10, 2025

changes
Ilia Kulikov
committed
5e9dd9a

Commits on Mar 11, 2025

changes
Ilia Kulikov
committed
9a39fca
changes
Ilia Kulikov
committed
18f609d
optional weight update group since we dont need it for stationary models like llm reward model
Ilia Kulikov
committed
211dc92
changes
Ilia Kulikov
committed
e2569a8

Commits on Mar 12, 2025

switch gsm8k rewards back to 0 and 1
Ilia Kulikov
committed
bf901b3

Commits on Mar 13, 2025

change batching logic in grpo loss
Ilia Kulikov
committed
500e28a
metric formatters added for grpo
Ilia Kulikov
committed
62d6c8e

Commits on Mar 18, 2025

more changes to math utils and rewards
Ilia Kulikov
committed
5bb5c06

Commits on Mar 19, 2025

allow original text in the example if needed
Ilia Kulikov
committed
d94ce26

Commits on Mar 24, 2025

microbatching wip
Ilia Kulikov
committed
3bbbe68

Commits on Mar 25, 2025

microbatching and validation implemented
Ilia Kulikov
committed
0c3f69f

Commits on Mar 26, 2025

safetensor loading fix
Ilia Kulikov
committed
13e49d3

Commits on Mar 28, 2025

validation during training, reference model offloading, global actor config
Ilia Kulikov
committed
33318b9

Commits on Apr 1, 2025

online dpo typo and small fixes
Ilia Kulikov
committed
3229a40
formatting
Ilia Kulikov
committed
42f6ee5
_add_ directive removed
Ilia Kulikov
committed
7bd4c0d

Commits on Apr 2, 2025

hf tokenizer support added
Ilia Kulikov
committed
20e3ede
default fix
Ilia Kulikov
committed
b4537ec

Commits on Apr 3, 2025

Jacklanchantin/online training (#1101)

authored
d8e45ec

Commits on Apr 5, 2025

grpo preset added
Ilia Kulikov
committed
e3996aa

Commits on Apr 10, 2025

type fix to allow none value
Ilia Kulikov
committed
cdb4c24

Commits on Apr 17, 2025

flexible sampling params
Ilia Kulikov
committed
2cb30f8

Commits on Apr 18, 2025

logit entropy track
Ilia Kulikov
committed
163f776
force v0 engine with newer vllm versions
Ilia Kulikov
committed
caa6541

Commits on Apr 24, 2025

make validation unit separately so that total examples counter is not reset at every validation
Ilia Kulikov
committed
d1db955

Commits on Apr 29, 2025

Commits on Apr 30, 2025

Add athene rm (#1154)
jacklanchantin
authored
5fb86ab
math-verify verifier added
Ilia Kulikov
committed
dd03a36

Commits on May 1, 2025

loss zeroer log, use prompt batch size in loss normalizer
Ilia Kulikov
committed
cf5703b
GRPO len norm support
Ilia Kulikov
committed
128dd3d
Add validation sampling params & force sync when starting train unit (#1156)
jacklanchantin
authored
0610d4e

Commits on May 2, 2025

check if self._step_nr exists when syncing (#1160)
jacklanchantin
authored
6ec348d

Commits on May 6, 2025

group DPO added
Ilia Kulikov
committed
aef5df4
different normalization
Ilia Kulikov
committed
edc8688
normalizer fix
Ilia Kulikov
committed
fd2b939
only force sync if sync_vllm_model_every_n_steps>0
jacklanchantin
committed
fba975f
Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
jacklanchantin
committed
1b50aad

Commits on May 8, 2025

only force sync if sync_vllm_model_every_n_steps>0 (#1169)
jacklanchantin
authored
33d9e00

Commits on May 9, 2025

use simple grad acc
Sainbayar Sukhbaatar
committed
869aa90
nll len norm flag
Sainbayar Sukhbaatar
committed
9744bb6
online dpo NLL len norm
Sainbayar Sukhbaatar
committed
182ab2c
typo on NLL len norm
Sainbayar Sukhbaatar
committed
3292773

Commits on May 21, 2025

Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
jacklanchantin
committed
45c8f8d
Remove Skywork-RM (#1180)
jacklanchantin
authored
effbf55
Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
jacklanchantin
committed
eaef36f
removing old math verification logic (#1181)

uralik
and
Ilia Kulikov
authored
f86dc67

Commits on May 22, 2025

Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
jacklanchantin
committed
b03a60d

Commits on May 23, 2025

replicas support added to vllm models (#1184)

uralik
and
Ilia Kulikov
authored
04c4064

Commits on May 27, 2025

stateful rollout bag in online recipe, microbatching in GRPO (#1185)

uralik
and
Ilia Kulikov
authored
1b77139

Commits on May 28, 2025

Commits on May 29, 2025

fix grad acc bug (#1191)
jacklanchantin
authored
ac40152

Commits on May 30, 2025

Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
jacklanchantin
committed
8e8043c

Commits on Jun 2, 2025

average and length normalized scores over multiple judgments
swarnadeep user
committed
9b0036e
Merge branch 'online_training' of github.com:facebookresearch/fairseq2 into online_training
swarnadeep user
committed
9a85388
Removing unused import
swarna
committed
e986ade

Commits on Jun 8, 2025

fix export keymap by reusing and reversing existing importing key map
Ilia Kulikov
committed
f78110b
typing style
Ilia Kulikov
committed
d588b52
sort
Ilia Kulikov
committed
457eb36
typing fix
Ilia Kulikov
committed
2a8f7f2
hg config export
Ilia Kulikov
committed
c075248

Commits on Jun 9, 2025

mypy
Ilia Kulikov
committed
666d9e5
styling
Ilia Kulikov
committed
9e235b9
mypy
Ilia Kulikov
committed
1a7187a
mypy
Ilia Kulikov
committed
845a8f5
mypy
Ilia Kulikov
committed
581c972
mypy
Ilia Kulikov
committed
f7e2297
mypy
Ilia Kulikov
committed
7ea63c3
raise keyerror if keep_jsonl_keys not in data file (#1201)
jacklanchantin
authored
76be659
mypy and refactor
Ilia Kulikov
committed
644d58e
mypy
Ilia Kulikov
committed
4ee2a5b
mypy
Ilia Kulikov
committed
a3c9aa4
style
Ilia Kulikov
committed
4fc88d4

Commits on Jun 11, 2025

r1 distilled llama asset added
Ilia Kulikov
committed
1d316b8
Merge branch 'main' into ot_merge
Ilia Kulikov
committed
72cd66d
Merge branch 'kulikov/qwen_export_fix' into ot_merge
Ilia Kulikov
committed
f0bae87

Commits on Jun 25, 2025

fixing
Ilia Kulikov
committed
627e3f6
Merge branch 'main' into ot_merge
Ilia Kulikov
committed
9bd4745
merge fix
Ilia Kulikov
committed
53f79c2
merge fix
Ilia Kulikov
committed
f3f24a8

Commits on Jun 27, 2025

refactoring
Ilia Kulikov
committed
598c3d2

Commits on Jul 1, 2025

dpo refactoring
Ilia Kulikov
committed
4bd9016
missing
Ilia Kulikov
committed
8dab75f
missing
Ilia Kulikov
committed
2778dea
remove old files
Ilia Kulikov
committed
f6a9fe4

Commits on Jul 2, 2025

more vllm engine args, qwen arch saver fix
Ilia Kulikov
committed
85cf9aa

Commits on Jul 3, 2025

moving to vllm worker extension class rather than separate class, enabling V1 engine
Ilia Kulikov
committed
df267e9

Commits on Jul 7, 2025

Support for pairwise judges in online training (#1194)

authored
a7ffaa5

Commits on Jul 8, 2025

run black formatter (#1225)
jacklanchantin
authored
1cc84ac
Add General Verifier for Math (#1224)
jacklanchantin
authored
14155b2

Commits on Jul 16, 2025

Jacklanchantin/general verifier fix (#1227)
jacklanchantin
authored
c362f7d

Commits on Jul 22, 2025

Merge branch 'main' into ot_merge
Ilia Kulikov
committed
cbdc0eb
v0.5 breaking change
Ilia Kulikov
committed
af1e84e

Commits on Jul 23, 2025

fix qwen parameter name converter, merge leak
Ilia Kulikov
committed
69351e3

Commits on Jul 25, 2025

Commits on Jul 28, 2025

converter fetch on recipe level (#1236)

uralik
and
Ilia Kulikov
authored
b8bbbba

Commits on Jul 29, 2025

restricting sampling in vllm with qwen models to avoid OOVs (#1237)

uralik
and
Ilia Kulikov
authored
1799948

Commits on Aug 8, 2025

Merge branch 'main' into online_training
Ilia Kulikov
committed
dbf3a3e

Commits on Aug 27, 2025

parallel vllm worker init, online dpo ref score fix when bs>1 and reward is binary (#1262)

uralik
and
Ilia Kulikov
authored
221c83b

Commits on Oct 27, 2025

Add truncated importance sampling and DrGRPO args (#1394)

jacklanchantin
and
Jack Lanchantin
authored
606459b

Commits on Oct 29, 2025

set VLLM_ALLOW_INSECURE_SERIALIZATION=1 for newer vllm versions and add a comment for calling newer vllm generate function (#1396)
lydiadli
authored
9c70654

Commits on Nov 18, 2025

remove unused vllm library (Worker) (#1441)
jacklanchantin
authored
34fd1b3

Commits on Nov 19, 2025

do not throttle client-server port (#1452)
uralik
authored
7d68329