[rl] Add XPU support to upstream GRPO training pipeline

[songhappy]

Summary

Enable the full GRPO RL pipeline (trainer + vLLM generator + TorchStore weight sync) to run on Intel XPU via the standard train.py entry point, with no changes to core torchtitan code.

Changes

1. rl/__init__.py: Guard PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True behind PYTORCH_XPU_ALLOC_CONF check. On XPU, expandable_segments corrupts the oneDNN memory allocator's USM pointers, breaking all oneCCL collectives. Also makes vLLM imports lazy to prevent vllm_xpu_kernels C extensions from initializing a SYCL context in the trainer process.

2. actors/generator.py: Auto-select the vLLM attention backend on XPU. vLLM's XPU platform only supports FLASH_ATTN and TRITON_ATTN; passing FLEX_ATTENTION raises ValueError. Setting backend=None lets vLLM default to flash_attn on XPU.

3. actors/trainer.py: Apply XPU compatibility patches on init.

4. xpu_compat.py (new): XPU-specific monkey-patches that are idempotent no-ops on CUDA:

   • Disable max_autotune (avoids OUT_OF_RESOURCES on backward autotuning)
   • Force FORCE_USE_FLEX_ATTENTION kernel option (flex_decoding crashes on Q_LEN < 128)
   • Strip unsupported separate_full_blocks kwarg from create_block_mask

5. train.py: Bootstrap callable sets ZE_AFFINITY_MASK for XPU device isolation (detected via inherited env from launcher).

6. config_registry.py: Add rl_grpo_lora_qwen3_0_6b config using FSDP (dp_shard=2) + LoRA that works on both CUDA and XPU.

7. run_grpo_lora_xpu.sh: Example launcher script for Intel XPU.

Test plan

• 10-step GRPO training on 4 Intel Max 1550 GPUs (2 trainer + 2 generator) via python3 -m torchtitan.experiments.rl.train --module alphabet_sort --config rl_grpo_lora_qwen3_0_6b
• Validation reward improved 0.182 -> 0.351 (real RL signal)
• Generator throughput: ~1162-1191 tok/s prompt, ~335-354 tok/s decode
• Trainer throughput: ~2377-4009 tokens/sec (fwd_bwd)
• Clean checkpoint save + graceful shutdown
• No impact on CUDA path (all XPU patches gated behind torch.xpu.is_available() or env var checks)

[pytorch-bot]

The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:

• ciflow/8gpu
• ciflow/rl

Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows.