Skip to content

Jacklanchantin/normalized rewards #1161

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 44 commits into
base: main
Choose a base branch
from
Draft

Jacklanchantin/normalized rewards #1161

wants to merge 44 commits into from

Conversation

jacklanchantin
Copy link
Contributor

What does this PR do? Please describe:
A summary of the change or the issue that is fixed.

Fixes #{issue number}

Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.

Check list:

  • Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
  • Did you read the contributor guideline?
  • Did you make sure that your PR does only one thing instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

Ilia Kulikov and others added 30 commits February 22, 2025 01:36
online training components wip
4c9fb9e
wip
6995cb9
Merge branch 'main' into online_training
1a4f79a
wip
8ca6d71
Merge branch 'main' into online_training
d776eb5
wip changes
d9f3bf0
wip
bcf05f4
more
e915733
changes
5e9dd9a
changes
9a39fca
changes
18f609d
optional weight update group since we dont need it for stationary mod…
211dc92 
…els like llm reward model
changes
e2569a8
switch gsm8k rewards back to 0 and 1
bf901b3
change batching logic in grpo loss
500e28a
metric formatters added for grpo
62d6c8e
more changes to math utils and rewards
5bb5c06
allow original text in the example if needed
d94ce26
microbatching wip
3bbbe68
microbatching and validation implemented
0c3f69f
safetensor loading fix
13e49d3
validation during training, reference model offloading, global actor …
33318b9 
…config
online dpo typo and small fixes
3229a40
formatting
42f6ee5
_add_ directive removed
7bd4c0d
hf tokenizer support added
20e3ede
default fix
b4537ec
@jacklanchantin @cbalioglu
Jacklanchantin/online training (#1101)
d8e45ec 
* Add Skywork Reward Model

* reorder

* working checkpoint with skywork

* unify VllmReward class

* cleanup wrap text

* actually not quite working. needs dp gather and scatter

* add gather scatter for vllm rm

* working checkpoint

* working checkpoint

* updates

* Instantiate metric recorders on rank 0 only (#1072)

* fix gather bug

* cleanup

* comment

* add grpo (runnable, but not increasing rewards yet

* log outputs

* cleannup

* rename

* merge

* fixes

* testing online dpo after merge

* bug fix

* fixing merge errors

* fix bugs

* merged with online_training

* update grpo

* cleanup

* fix grpo bug

* cleanup

* cleanup

* isort/black

* move vllm generate_rewards to its own function

* refactor how reward models use prompt_batch

* remove breakpoint

* working chkpt

* remove irrelevant skywork stuff

---------

Co-authored-by: jacklanchantin user <[email protected]>
Co-authored-by: Can Balioglu <[email protected]>
grpo preset added
e3996aa
type fix to allow none value
cdb4c24
Ilia Kulikov and others added 14 commits April 17, 2025 18:51
flexible sampling params
2cb30f8
logit entropy track
163f776
force v0 engine with newer vllm versions
caa6541
make validation unit separately so that total examples counter is not…
d1db955 
… reset at every validation
@jacklanchantin
Record rollouts in logs, update num examples, separate train/valid in…
6b3284e 
… wandb (#1148)

* add log_rollouts function in _online_finetune/_common.py

* set default False

* add update_batch_metrics

* add update_batch_metrics to grpo

* update_batch_metrics in validation

* wandb separate train/valid

* remove comment
@jacklanchantin
validate before training (#1153)
dd4ed22 
* add bool to validate before training

* remove blank line

* set online training preset: validate_step_0 = True

* validate_step_0 --> validate_before_training
@jacklanchantin
Add athene rm (#1154)
5fb86ab 
* add athene RM

* remove local path

* docstrings

* docstrings

* default tokenzier

* remove divpo code

* add handler

* add new vllm args

* remove valid_n
math-verify verifier added
dd03a36
loss zeroer log, use prompt batch size in loss normalizer
cf5703b
GRPO len norm support
128dd3d
@jacklanchantin
Add validation sampling params & force sync when starting train unit (#…
0610d4e 
…1156)

* add validation sampling params. force sync when restoring

* force sync vllm in _maybe_restore_state

* new frorce sync

* try execpt

* raise error
@jacklanchantin
check if self._step_nr exists when syncing (#1160)
6ec348d
@jacklanchantin
change var name
18dace7
@jacklanchantin
check if self._step_nr exists when syncing (#1160)
c6fc816
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cbalioglu cbalioglu Awaiting requested review from cbalioglu cbalioglu will be requested when the pull request is marked ready for review cbalioglu is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jacklanchantin @cbalioglu @facebook-github-bot