Relax supports multiple policy gradient algorithms, all selected via the --advantage-estimator flag. This document covers PPO and the primary GRPO-family algorithms (for On-Policy Distillation, see the dedicated page).
GRPO, Score Centering, RLOO, CISPO, GSPO, SAPO, and M2PO share the same actor/rollout service topology, although RLOO is synchronous-only and enforces fixed batch invariants. PPO additionally requires a Critic model and an Advantages service; start from the PPO training recipe instead of only replacing GRPO_ARGS.
REINFORCE++ and REINFORCE++-baseline also reuse the GRPO service topology, but their return, global normalization and KL contracts are algorithm-specific. See REINFORCE++ Training before enabling either estimator.
GRPO (Group Relative Policy Optimization) is the default algorithm in Relax. It broadcasts the group-relative scalar reward to every token and uses a standard PPO-Clip objective.
Reference: DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
The GRPO objective is the standard PPO-Clip:
where
| Parameter | Default | Description |
|---|---|---|
--advantage-estimator grpo |
default | Enable GRPO |
--eps-clip |
0.2 |
Clipping margin (ratio range = [1-ε, 1+ε]) |
--eps-clip-high |
same as --eps-clip |
Upper clipping margin; can be set differently for asymmetric clipping |
--clip-grad |
— | Gradient clipping norm |
GRPO is the default algorithm — no parameter changes needed. Just run the training script directly:
MODEL_DIR=/path/to/model \
DATA_DIR=/path/to/data \
EXP_DIR=/path/to/exp \
bash scripts/training/text/run-qwen3-4B-8xgpu.shScore Centering corrects policy-gradient drift when the rollout distribution differs from the training distribution, for example when the sampler uses lower precision. Here, the score means the gradient of the policy log-probability, not the reward. SC subtracts the expected score under the generation distribution; it does not make rollout and training log-probabilities identical.
Relax records generation-time top-K candidate IDs and probabilities, scores the same candidates in the trainer, and approximates the unrecorded tail. The default advantage is the response reward minus the prompt group's mean reward, without division by the group standard deviation. Unlike RLOO, this mean includes the current response. The loss uses score centering rather than PPO clipping.
For an existing text-only Megatron recipe, replace its algorithm arguments with the following block and pass "${SC_ARGS[@]}" to the training command:
SC_ARGS=(
--loss-type policy_loss
--advantage-estimator score_centering
--calculate-per-token-loss
--n-samples-per-prompt 8
--rollout-temperature 1.0
--rollout-top-p 1.0
--rollout-top-k -1
--rollout-top-logprobs-num 128
--score-centering-is none
--kl-coef 0
)Keep reward normalization enabled and remove --disable-rewards-normalization, --normalize-advantages, OPD, and outer TIS settings from the original recipe. SC requires at least two responses per prompt and global valid-token loss normalization. --eps-clip and --eps-clip-high do not control this loss. Reference-policy KL loss and entropy regularization remain separate options.
| Parameter | Parser default | Description |
|---|---|---|
--rollout-top-logprobs-num |
128 |
Candidate count K, from 1 to the real vocabulary size; larger values increase collection, transfer, and scoring costs without changing sampling top-k |
--score-centering-is |
none |
Internal importance weighting: none, tis, or mis, applied to both the sampled and centering terms |
--score-centering-tis-clip |
2.0 |
Positive finite weight upper bound for TIS |
--score-centering-mis-low |
0.5 |
Lower bound of the MIS retained-weight interval |
--score-centering-mis-high |
5.0 |
Upper bound of the MIS interval; bounds must satisfy 0 < low <= high |
To combine SC with TIS, use --score-centering-is tis --score-centering-tis-clip 2.0. Do not also enable --use-tis or --custom-tis-function-path. SC also rejects --use-opsm, --use-unbiased-kl, and a custom policy-loss reducer.
Custom rollout functions must preserve the sampled behavior log-probabilities and aligned rollout_topk_token_ids / rollout_topk_log_probs, including loss masks for tool observations. Returned probabilities must reflect the actual sampling temperature and filters. Top-p/min-p filtering requires a finite sampling top-k; top_p < 1 additionally requires --sglang-sampling-backend pytorch. Non-unit temperature requires SGLANG_RETURN_ORIGINAL_LOGPROB=0. Speculative decoding, sglang_rl_on_policy_target, and filtered sampling on the Ascend backend are not supported by this capture path. SC uses the Megatron text-token policy loss, not the FSDP diffusion training path.
SC reports token-mask-aware means under train/:
sc_rollout_head_mass/sc_train_head_mass: probability mass covered by the recorded candidates in each distribution.sc_tail_ratio/sc_tail_floor_fraction: tail-mass ratio and the fraction of positions where either tail is below the fixed1e-6numerical floor.sc_correction: the signed scalar centering term; its magnitude alone does not measure training quality.sc_importance_weight,sc_is_clip_fraction, andsc_zero_weight_fraction: internal weighting diagnostics. With--score-centering-is none, these are 1, 0, and 0.
--get-mismatch-metrics adds mismatch diagnostics without applying extra TIS weighting. Interpret these alongside validation reward, response length, truncation rate, and gradient norm; successful SC does not require mismatch to converge to zero.
Use the eight-GPU Qwen3-0.6B Countdown script. First prepare the BF16 model, its FP8 conversion, and Countdown JSONL data following the setup instructions; this recipe also requires the SGD and BF16 export fixes listed there. Submit from the repository root to a dedicated Ray cluster with eight GPUs:
MODEL_DIR=/path/to/models \
DATA_DIR=/path/to/countdown-data \
SAVE_DIR=/path/to/checkpoints \
RAY_ADDRESS=http://ray-head:8265 \
USE_SC=1 USE_TIS=0 \
bash scripts/entrypoint/ray-job.sh \
examples/algorithms/score-centering/run-qwen3-0.6B-8xgpu.shRAY_ADDRESS is the Ray Jobs HTTP address. USE_SC=1 USE_TIS=0 selects SC; set USE_TIS=1 for SC with internal TIS. Setting USE_SC=0 selects native RLOO, with optional native TIS controlled by the same switch. The script defaults to USE_SC=0 USE_TIS=0, so explicitly set both switches when selecting an experiment.
This recipe uses a BF16 trainer, an FP8 sampler, SGD with learning rate 0.01, and 400 updates. These are experiment settings; SC does not require SGD. Online evaluation uses the FP8 sampler. Use a new EXP_NAME for each independent run; full SGD optimizer resume is currently limited as described in the setup instructions.
RLOO (REINFORCE Leave-One-Out) uses the other samples for the same prompt as an unbiased baseline. Relax implements synchronous RLOO with an unclipped REINFORCE policy loss; it does not use PPO ratios or clipping.
Reference: Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs.
For a prompt with
Unlike GRPO, RLOO does not divide by the group standard deviation. The token loss is:
Each sample's scalar advantage is broadcast to its response tokens. Relax masks padding and normalizes the summed loss by the global number of valid response tokens
This global-token reduction does not apply a separate train/pg_clipfrac is always 0 because RLOO uses no clipping.
| Parameter | Requirement | Description |
|---|---|---|
--advantage-estimator rloo |
required | Enable RLOO |
--n-samples-per-prompt |
at least 2
|
Group size |
--rollout-batch-size × --n-samples-per-prompt |
equals --global-batch-size
|
Exactly one optimizer update per rollout |
--num-steps-per-rollout |
unset or 1
|
Reusing the same rollout for multiple unclipped updates is rejected |
--calculate-per-token-loss |
enabled | Use global valid-token normalization; per-response token means would reweight unequal-length responses by |
--kl-coef |
0 |
Reward-side KL shaping is not implemented for RLOO; with a valid --ref-load <checkpoint>, use --use-kl-loss --kl-loss-coef <value> for the supported direct KL penalty |
--max-staleness |
0 |
Stale rollouts are rejected because the unclipped objective has no importance-ratio correction |
| reward normalization | enabled | RLOO's group transformation runs in the normalized-reward path |
--normalize-advantages |
disabled | Post-DP whitening would change RLOO semantics and make results partition-dependent |
--fully-async, --hybrid, --partial-rollout, --use-dynamic-global-batch-size
|
disabled | RLOO currently requires synchronous, fixed-size rollout batches |
The batch sizes are hardware-tunable as long as their equality is preserved. For example, ROLLOUT_BATCH_SIZE=4, N_SAMPLES=8, and GLOBAL_BATCH_SIZE=32 retain one update per rollout while reducing per-step memory relative to 16 × 8 = 128.
Training rollout logs publish the following final metric names:
rollout/rloo/baseline_mean: mean LOO baseline (equal to the mean group reward; retained as an explicit baseline trace)rollout/rloo/adv_abs_mean: mean absolute RLOO advantagerollout/rloo/no_signal_frac: fraction of effective loss tokens attached to zero-advantage samplesrollout/rloo/empty_response_frac: fraction of samples with a literally empty responserollout/rloo/zero_adv_group_frac: fraction of complete groups with zero advantages throughoutrollout/rloo/dropped_group_frac: fraction of observed groups omitted from diagnostics because their size is incomplete
These diagnostics are training-only, purely observational rollout statistics; they do not affect the training path. Evaluation uses its own sampling group size and does not emit misleading eval/*/rloo/* values. They are also omitted when a custom reward post-processor or agentic custom-advantage hook replaces the standard RLOO signal, because raw rewards cannot reconstruct the optimizer input in those modes.
Use the dedicated Qwen3-0.6B GSM8K recipe. Its batch and rollout settings can be overridden with environment variables:
MODEL_DIR=/path/to/models \
DATA_DIR=/path/to/data \
NUM_ROLLOUT=60 \
ROLLOUT_BATCH_SIZE=4 \
N_SAMPLES=8 \
GLOBAL_BATCH_SIZE=32 \
bash examples/algorithms/run-qwen3-0.6B-1xgpu-rloo.shSet ADVANTAGE_ESTIMATOR=grpo to run a control arm with the same recipe and seeds. The recipe writes normalized GSM8K data to a writable artifact cache (override with RLOO_DATA_CACHE_DIR) and appends an instruction to emit the final answer as \boxed{...}, matching the math reward parser contract.
PPO (Proximal Policy Optimization) is an actor-critic algorithm. Relax trains a separate Critic to predict token-level values, computes GAE advantages and returns, applies PPO-Clip to the Actor, and applies clipped value loss to the Critic.
Reference: Proximal Policy Optimization Algorithms.
The temporal-difference residual and GAE recursion are:
The Actor then uses the same clipped policy objective shown for GRPO, but with Critic-derived token-level advantages. The Critic minimizes the maximum of clipped and unclipped squared value errors.
| Parameter | Default | Description |
|---|---|---|
--advantage-estimator ppo |
— | Enable PPO and the Critic service graph |
--gamma |
1.0 |
GAE discount factor |
--lambd |
1.0 |
GAE lambda |
--eps-clip |
0.2 |
Actor clipping margin |
--value-clip |
0.2 |
Critic value clipping range |
--num-critic-only-steps |
0 |
Initial Critic-only warmup steps |
--critic-lr |
same as --lr |
Critic learning rate |
PPO cannot be enabled by changing only the algorithm argument because its service graph requires critic and advantages resources. Fully-async PPO is not currently supported; use the dedicated synchronous colocate recipe:
MODEL_DIR=/path/to/models \
DATA_DIR=/path/to/data \
EXP_DIR=/path/to/experiments \
bash scripts/training/text/run-qwen35-9B-8xgpu-ppo.shSee PPO Training for the resource topology, checkpoint rules, and KL constraints.
CISPO (Clipped Importance-ratio Soft Policy Optimization) preserves gradient signal for out-of-trust-region tokens instead of zeroing it out. It caps gradient magnitude via a stop-gradient'd coefficient while keeping the gradient direction alive.
Reference: MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention.
The CISPO objective is:
where
and
| Parameter | Default | Recommended | Description |
|---|---|---|---|
--advantage-estimator cispo |
— | — | Enable CISPO |
--eps-clip |
0.2 |
0.2 |
Lower clipping margin (ratio lower bound = 1 - eps_clip) |
--eps-clip-high |
same as --eps-clip |
10 |
Upper clipping margin (ratio upper bound = 1 + eps_clip_high). Set to 10 to effectively unclamp the upper side |
--kl-loss-coef |
0.0 |
0.001 |
KL loss coefficient. Recommended: 0.001 to add a small KL penalty that constrains policy drift |
--use-kl-loss |
off | on | Enable KL loss computation (required for --kl-loss-coef to take effect) |
--use-tis |
off | on | Token Importance Sampling — recommended to enable with CISPO |
--clip-grad |
— | 1.0 |
Gradient clipping norm |
Use any existing GRPO training script and replace GRPO_ARGS with CISPO_ARGS:
CISPO_ARGS=(
--advantage-estimator cispo
--use-kl-loss
--kl-loss-coef 0.001
--eps-clip 0.2
--eps-clip-high 10
--use-tis
)GSPO (Group-wise Sequence-level Policy Optimization) differs from GRPO in how KL divergence is computed: GSPO uses sequence-level KL instead of per-token KL. Every token in a sequence shares the same KL value (the mean over all tokens in that sequence), providing uniform constraint strength within a sequence.
GSPO uses the same PPO-Clip objective as GRPO, but the ratio is computed from sequence-level KL:
Every token's ratio is
| Parameter | Default | Description |
|---|---|---|
--advantage-estimator gspo |
— | Enable GSPO |
--eps-clip |
0.2 |
Clipping margin |
--eps-clip-high |
same as --eps-clip |
Upper clipping margin |
--clip-grad |
— | Gradient clipping norm |
GSPO_ARGS=(
--advantage-estimator gspo
--eps-clip 0.2
)SAPO (Soft Adaptive Policy Optimization) replaces hard clipping with a smooth sigmoid gate. The gate's steepness is controlled by a temperature parameter, implementing a differentiable trust region constraint.
SAPO's core is a sigmoid gate centered at ratio=1:
where
-
$A > 0$ : use$\tau_\text{pos}$ (default 1.0) -
$A \leq 0$ : use$\tau_\text{neg}$ (default 1.05, stronger suppression for negative tokens)
SAPO objective:
| Parameter | Default | Description |
|---|---|---|
--advantage-estimator sapo |
— | Enable SAPO |
--sapo-tau-pos |
1.0 |
Temperature for positive advantages |
--sapo-tau-neg |
1.05 |
Temperature for negative advantages (higher = stronger suppression) |
--clip-grad |
— | Gradient clipping norm |
SAPO_ARGS=(
--advantage-estimator sapo
--sapo-tau-pos 1.0
--sapo-tau-neg 1.05
)M2PO (Second-Moment Trust Policy Optimization) uses the second moment of the log importance ratio over harmful tokens as its trust-region constraint: it tightens clipping only when that second moment exceeds a budget, and keeps the token otherwise. Compared to fixed clipping, it retains more useful gradient and mitigates entropy collapse in off-policy (stale-data) regimes, making it purpose-built for mini-batch reuse and asynchronous training.
Reference: Prosperity before Collapse: How Far Can Off-Policy RL Reach with Reuse of Mini-Batch Data? (NeurIPS 2025).
M2PO only constrains the "harmful" tokens that PPO would clip — those whose advantage sign aligns with the ratio's drift and would cause an over-update:
where $r_t = \exp(-\text{KL}t)$ and $\text{KL}t = \log\pi{\theta\text{old}}(o_t) - \log\pi_\theta(o_t)$. The second moment of the log-ratio over these tokens is:
Relax solves this statistic independently on each Megatron microbatch's local
tokens. Logging preserves the existing aggregation: with
--calculate-per-token-loss, train/ppo_kl_m2_before and
train/ppo_kl_m2_after are weighted by the microbatch's loss-token count;
otherwise, each microbatch scalar is summed unchanged and the framework
divides by the sample count. The latter is not a sample-weighted mean. Neither
mode pools all harmful tokens in the global batch. These logged diagnostics
do not feed back into the loss.
- If
$M_2 \le$ kl2_budget: no clipping, all tokens are kept; - Otherwise, use water-filling to select the previous observed breakpoint. This gives a conservative trust-region radius
$\tau$ whose capped sum does not exceed$|\mathcal{H}| \cdot \text{kl2_budget}$ , yielding the clip band$[e^{-\tau},\ e^{\tau}]$ .
The final clipping margin is ppo_kl_m2_after is the legacy solver diagnostic for the selected breakpoint before that floor is applied; in the first-breakpoint edge case it retains the uncapped local mean. It is not a recomputation after the final policy clip. The policy loss reuses the PPO-Clip pessimistic form from the GRPO section, with only the clip bounds solved adaptively.
| Parameter | Default | Recommended | Description |
|---|---|---|---|
--advantage-estimator m2po |
— | — | Enable M2PO |
--m2po-kl2-budget |
0.01 |
0.01–0.04 |
Second-moment budget per harmful token. Smaller = tighter/more-frequent clipping, larger = more off-policy tolerance (the paper uses 0.04) |
--m2po-miniclip-low |
0.3 |
0.2 |
Lower-side clip-margin floor (the margin is at least miniclip_low) |
--m2po-miniclip-high |
0.5 |
0.28 |
Upper clip-margin floor |
--use-rollout-logprobs |
off | on for stale async data | Use the behavior-policy log probabilities that generated each token as M2PO's old policy |
--use-tis |
off | off with rollout log probs | TIS is a post-loss correction and does not drive M2PO's adaptive threshold; it is mutually exclusive with --use-rollout-logprobs |
M2PO derives its clip bounds adaptively, so it does not use
--eps-clip/--eps-clip-high.
::: warning Existing implementation behavior
The registry refactor preserves M2PO's existing computation: reward processing
passes through raw sample rewards, and the threshold solver does not filter
tokens by the loss mask or aggregate its statistic across CP ranks. Its
breakpoint comparisons still use host scalars. Registration does not establish
equivalence to the paper. Because the clipping statistics are local, M2PO
declares supports_context_parallel=False: startup requires
--context-parallel-size 1 and dynamic context parallelism disabled. This
validation leaves the algorithm's reward, solver and metric calculations
unchanged.
:::
M2PO's benefit grows with how off-policy the training data is, so reach for it first in these scenarios:
- Asynchronous training with large staleness: in fully-async mode the rollout weights lag noticeably behind the actor (
--max-stalenessof 32, 256, or higher), and stale samples inflate the importance ratio. Fixed clipping then either zeroes out many tokens (losing gradient) or lets them through (causing over-updates); M2PO uses the second moment to adaptively tighten only the genuinely "harmful" fraction, suppressing collapse while preserving the learning signal. - Mini-batch reuse / multi-step sampling: when the same rollout batch is reused across several update steps, the later steps effectively train on off-policy data too, and M2PO extends the usable lifetime of that batch.
- Late-training entropy collapse or reward stagnation: when fixed clipping narrows the policy too quickly and starves exploration, M2PO's looser adaptive bounds help sustain entropy and delay collapse.
Conversely, under strictly on-policy synchronous training (--max-staleness 0 with per-step weight sync), M2PO's gain over GRPO is limited — start from GRPO as a baseline there.
M2PO is valid in true-on-policy mode, but then its importance ratio is exactly
one and adaptive clipping does not engage. To evaluate its stale-data behavior,
make sure the run supplies old-policy log probabilities rather than
auto-enabling --true-on-policy-mode. For fully-async rollout staleness, pass
--use-rollout-logprobs: --use-tis is applied only after M2PO has already
chosen its clip bounds.
Use any existing GRPO training script and replace GRPO_ARGS with M2PO_ARGS:
M2PO_ARGS=(
--advantage-estimator m2po
--m2po-kl2-budget 0.01
--m2po-miniclip-low 0.2
--m2po-miniclip-high 0.28
)| Algorithm | Advantage Computation | Policy Loss | KL Constraint |
|---|---|---|---|
| PPO | Critic values + GAE | PPO-Clip (hard clip) | Disabled in the current synchronous topology |
| GRPO | Group-relative reward | PPO-Clip (hard clip) | Optional KL loss |
| Score Centering | Group-mean-centered reward, without group-standard-deviation scaling | Centered score with optional internal TIS/MIS | Reward-side KL disabled; optional direct KL loss |
| REINFORCE++ | Token KL-to-go return + global token normalization | PPO-Clip (hard clip) | k1 KL in shaped reward |
| REINFORCE++-baseline | Inclusive group mean + global token normalization | PPO-Clip (hard clip) | Separate k2 KL loss |
| CISPO | Group-relative reward | Stop-gradient coefficient | Recommended KL loss |
| GSPO | Group-relative reward | PPO-Clip + sequence-level KL | Sequence-level ratio |
| SAPO | Group-relative reward | Sigmoid gate | Temperature-controlled |
| M2PO | Raw sample reward broadcast to tokens | Adaptive second-moment clip | Optional KL loss (favor for large-staleness / off-policy) |
| RLOO | Leave-one-out baseline | Unclipped REINFORCE | Optional KL loss (same as GRPO) |