Document how to change the training objective by subclassing a trainer

[sergiopaniego]

What does this PR do?

Adds a Change the training objective section to the training customization guide.

Every section in customization.md today customizes training by passing an argument: optimizers and schedulers, an 8-bit reference model, callbacks, evaluation metrics, mixed precision, gradient accumulation. The guide says nothing about the case where the objective itself differs, even though subclassing a trainer and overriding compute_loss is a pattern the library relies on throughout:

• GKDTrainer and GOLDTrainer extend SFTTrainer
• GMPOTrainer, MiniLLMTrainer and GRPOWithReplayBufferTrainer extend GRPOTrainer
• XPOTrainer and NashMDTrainer extend OnlineDPOTrainer
• examples/sft_diffusion_gemma/sft_diffusion_gemma.py extends SFTTrainer with a block-diffusion objective, entirely from an example script

Today the only traces of this in the docs are a one-line pointer in sft_trainer.md and a row in example_overview.md.

The new section documents the three override points (compute_loss, the config, the collator plus _prepare_dataset), then links to the block-diffusion example as a complete in-repo case and to one external project using the same pattern.

Note on scope: this documents subclassing a public trainer for a small, local delta, which is what GMPO, GKD, GOLD, XPO and NashMD do. It does not touch the self-contained trainer route on _BaseTrainer described in AGENTS.md, which remains the way to add a full new method.

Docs only, no code changes.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed.

---

Note

Low Risk
Documentation-only change with no runtime or API impact.

Overview
Adds a Change the training objective section to the training customization guide (customization.md), filling a gap where other sections only cover config/callback-style tweaks.

The new content explains subclassing a trainer (e.g. DPOTrainer) and overriding compute_loss, with a concrete hinge-loss DPO example using selective_log_softmax. It also covers extending DPOConfig for custom hyperparameters, supplying a custom collator and skipping default dataset prep via _prepare_dataset, and points readers to the in-repo block-diffusion SFT example and the external Antidoom/Antislop DPOTrainer extension as end-to-end references.

^{Reviewed by Cursor Bugbot for commit 8710f4a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

can you please remove this? internally, we're actually trying to do the opposite internally.

What you can say instead is that it's the simplest way to customize training, something like this.

[qgallouedec]

super useful, thanks, a few modifications needed

[qgallouedec]

same here, please remove this line

[qgallouedec]

data_collator is an argument of DPOTrainer, we shouldn't need to override the init.

[sergiopaniego]

updated, thanks for the review! @qgallouedec

[qgallouedec]

thanks, all good now.