[codex] fix data indexing and clip metrics

[yifanzhang-pro]

Summary

This is a narrower replacement for the useful parts of #23.

• Preserve the existing README fix on main that points installation at requirements.txt.
• Update process-data.py to write 0-based repeat indices into the schema consumed by RLHFDataset: extra_info["index"].
• Fix GRPO score aggregation to use torch.stack, preserving tensor dtype/device and computing the standard deviation over the response group.
• Make REINFORCE hard-clamp clip fraction metrics report how often w falls outside the clamp bounds instead of always reporting zero.

Notes

The GRPO advantage change affects the training signal because it fixes the normalization used before policy loss computation.

The REINFORCE pg_clipfrac / pg_clipfrac_lower change only affects logged metrics. It does not change A, pg_losses, or pg_loss.

Validation

• python -m py_compile process-data.py verl/trainer/ppo/core_algos.py
• git diff --check
• uvx ruff check process-data.py verl/trainer/ppo/core_algos.py

Not run: full torch/pandas tests in this temporary checkout because the local Python environment lacks project dependencies such as torch and pandas.

[yifanzhang-pro]

/gemini review