[Exploratory] Drop accelerate launch: the CLI, docs and examples launch with torchrun

[qgallouedec]

Why

Important

This does not drop support for accelerate launch. accelerate stays a dependency and accelerate launch train.py keeps working, with its config files. What changes is the default: the launcher the trl CLI uses and the one the docs and examples show is torchrun.

For multi-GPU CUDA, accelerate launch is a wrapper around torch.distributed.run: it reads a config file, sets ACCELERATE_* env vars, then calls torchrun. What it adds on top:

• a config file (accelerate config) holding num_processes, mixed precision and the DeepSpeed/FSDP plugin settings
• device auto-detection (all GPUs by default, and CUDA / XPU / NPU / TPU / MPS / CPU backends)
• other launchers: TPU xla_spawn, DeepSpeed pdsh multi-node, MPI, SageMaker, notebook_launcher

For a Trainer-based library none of this is needed: TrainingArguments already builds the DeepSpeed and FSDP plugins from --deepspeed / --fsdp / --bf16, and TRL only supports CUDA in practice. So TRL users end up learning a second config format (accelerate yaml) that duplicates the training arguments, and users who know torchrun have to guess how --num_processes maps to --nproc_per_node. Launching with torchrun directly removes that layer and makes the CLI, docs and examples say the same thing as the rest of the PyTorch ecosystem.

What does this PR do?

trl <cmd> now launches its training script with torchrun instead of accelerate launch, and every launch command in the docs and examples follows.

• trl/cli/commands/training.py: arguments the script does not know go to torchrun (--nproc_per_node, --nnodes, --master_port, ...), from the command line or from the --config YAML. Defaults to one process per accelerator. With no launcher argument and a single device, the script runs as python script.py (what accelerate launch did too), so nothing opens a rendezvous port.
• Deleted trl/cli/accelerate_launcher.py, trl/cli/accelerate_config.py, --accelerate_config and the shipped trl/accelerate_configs/*.yaml. DeepSpeed and FSDP are configured through the training arguments like any transformers.Trainer: --deepspeed zero3.json, --fsdp [--fsdp_config fsdp2.json].
• examples/accelerate_configs/ becomes examples/deepspeed_configs/zero{1,2,3}.json and examples/fsdp_configs/fsdp2.json (same settings as the accelerate plugin defaults: auto batch sizes, bf16 auto, ZeRO-3 gathers 16-bit weights on save).
• alst_ulysses_4gpu.yaml and context_parallel_2gpu.yaml had no user and are gone. The 1M-context example sets fsdp=True, parallelism_config=ParallelismConfig(cp_size=8) in its SFTConfig instead of the yaml (parallelism_config is not CLI-parsable).
• trl env no longer prints the accelerate default config.
• tests/distributed: torchrun --nproc_per_node 2 ... --deepspeed/--fsdp with minimal JSON configs.

Behaviour changes

• ~/.cache/huggingface/accelerate/default_config.yaml is no longer read by trl.
• No more TPU/MPS launch path in the CLI.
• FSDP2 through --fsdp_config needs transformers >= 4.57 (fsdp_version key); with 4.56.2 (our floor) accelerate launch still works for it since accelerate stays a dependency.
• torch.accelerator.device_count() for the default process count needs torch >= 2.6.
• Docs: bare accelerate launch train.py became torchrun --nproc_per_node 8 train.py; CUDA_VISIBLE_DEVICES=n accelerate launch became python. The Online DPO benchmark commands stay as they were (they reproduce from a v1.10.0 checkout).

Tested

2xH100 via torchrun, tiny Qwen2, 3 steps: SFT ddp / zero2 / zero3 / fsdp2, DPO zero3, and SFT with FSDP2 + ParallelismConfig(cp_size=2). tests/test_cli.py (single-process path + new test_sft_torchrun), tests/test_cli_utils.py, tests/test_examples_index.py. tests/distributed/test_distributed.py on 2 GPUs: 35 passed, 12 skipped (liger not installed).

Untested: the DeepSpeed Ulysses (sp_size) snippet in the long-context guide, and multi-node.