RL Stack Refactoring: Call for Contributions

[s1lent4gnt]

RL Stack Refactoring — Call for Contributions

We have been working on a major RL stack refactoring in LeRobot. The goal is to build a solid RL foundation that makes reinforcement learning and fine-tuning VLAs with RL as easy as imitation learning already is — and to make adding new RL algorithms straightforward for the community.

The first RL algorithm in LeRobot — HIL-SERL (RLPD/SAC with actor-learner architecture, reward classifier, and human interventions) — is working (#504, PR #644). Now we want to:

1. Solidify the foundations and tighten the interfaces
2. Make adding new RL algorithms easy — one algorithm, one file
3. Support VLA fine-tuning with RL (ConRFT, QC-FQL, RECAP, etc)
4. Integrate pluggable reward models for both IL and RL

We welcome community feedback, contributions, and ideas. You don't need to write extensive code on your own — any input on a sub-component or comment, however small, is appreciated.

Coordination: Discord #reinforcement-learning

---

Current Architecture

This architecture is introduced in the RL refactoring PR. The RL stack lives in src/lerobot/rl/ and follows LeRobot's existing patterns:

src/lerobot/rl/
├── algorithms/
│   ├── base.py          # RLAlgorithm ABC + RLAlgorithmConfig (draccus.ChoiceRegistry)
│   └── sac.py           # SAC training implementation
├── trainer.py           # RLTrainer (orchestrates training steps)
├── buffer.py            # ReplayBuffer
├── data_sources/
│   └── data_mixer.py    # DataMixer + OnlineOfflineMixer
├── learner.py           # Learner process (GPU training)
├── actor.py             # Actor process (environment interaction)
├── learner_service.py   # gRPC service for distributed training
└── ...

Key design decisions already in place:

• RLAlgorithmConfig uses draccus.ChoiceRegistry — same pattern as policies, cameras, motors
• RLAlgorithm.update(batch_iterator) — algorithms consume from an iterator, controlling batch consumption (e.g. UTD ratio)
• Actor-Learner architecture — distributed via gRPC, actors collect experience, learner trains
• DataMixer — mixes heterogeneous data sources (OnlineOfflineMixer mixes offline + online demonstrations)

---

Roadmap

Phase 1: Solidify Foundations (SAC stays working, interfaces tighten)

• Make RLAlgorithmConfig a first-class explicit config (decouple from policy_cfg)
• Clean up RLAlgorithm public API (select_action, update, get_weights/load_weights, configure_data_iterator)
• Improve documentation and add an RL training tutorial
• Add tests for the core RL components

Phase 2: Pluggable Reward Models

• Create src/lerobot/rewards/ package — shared reward models for both IL and RL
• Migrate existing reward classifier from policies/sac/reward_model/
• Migrate SARM from policies/sarm/ (see SampleWeighter refactor for related IL abstraction)
• Support zero-shot reward models (TOPReward, VITA)
• Papers: SARM, TOPReward, VITA

Phase 3: VLA Fine-Tuning Algorithms

• RECAP — Advantage-conditioned policies for heterogeneous data. Paper: π*0.6. Community: lerobot_recap, PR #2923
• QC_FQL - Flow Q-learning (FQL) agent with action chunking (QC). Community: PR #1818
• ConRFT — Unified offline-online consistency policy. Community: PR #1823
• DSRL (low complexity) — Steer frozen diffusion policy via latent-noise SAC; builds on existing SAC infra. Paper: arXiv:2506.15799

---

Open for Ideas

We want to hear from the community:

• What would make your life easier when training RL policies in LeRobot?
• What RL algorithms are you most excited about for robot learning?
• What other RL-based VLA fine-tuning methods should we support?
• What reward model approaches have worked best for your tasks?

Please comment below or open a PR. Contributions are highly encouraged!

Your help will make LeRobot a powerful and accessible framework for robot RL 🤖

[neo-alex]

Hey @s1lent4gnt, this RL refactoring sounds like a very exciting improvement - especially the support for VLA fine-tuning with RL. Really looking forward to it 👍

What would make your life easier when training RL policies in LeRobot?

--> I think it is already part of your roadmap, but what would be really great is to decouple as much as possible the RL algorithms (e.g. SAC or its fancier HIL-SERL version) from their policy/critic neural net definition: for instance, when training HIL-SERL, this would easily enable reusing a (pretrained) VLA of our choice instead of starting from scratch with a hardcoded model like today.

Thanks!

[sezan92]

Really interested to start working on it

[s1lent4gnt]

Hey @neo-alex, thanks for the feedback!

--> I think it is already part of your roadmap, but what would be really great is to decouple as much as possible the RL algorithms (e.g. SAC or its fancier HIL-SERL version) from their policy/critic neural net definition: for instance, when training HIL-SERL, this would easily enable reusing a (pretrained) VLA of our choice instead of starting from scratch with a hardcoded model like today.

Yes this is in our roadmap.

[jackvial]

What would make your life easier when training RL policies in LeRobot?

@s1lent4gnt Improved documentation and first class support for so101 in sim and on real hardware would help a lot, here's some of notes on the current RL docs and some suggestions for improvements that I think might make getting up and running with RL in LeRobot a bit a smoother.

High-Level Feedback

The current RL documentation is centered around HIL-SERL, I think HIL-SERL should instead be treated as a special case of RL rather than the primary entry point.

More concrete examples in the docs and following more of a tiered progression could help a lot. Users should be able to start with a minimal working example and gradually move toward more advanced setups like deploying on real hardware and HIL training.

---

Proposed RL Documentation Structure

1. Run a Pretrained RL Policy in Simulation

Provide the simplest possible entry point.

Goal:

• Run a pretrained RL policy for an SO101 arm in simulation
• Should ideally be a single command
• Get quick feedback and confirm that the environment and dependencies are set up correctly for inference

Example workflow:

• Download pretrained model
• Launch simulator
• Run inference

This step should require no training or configuration.

Purpose:

• Fast early feedback and context setting
• Reduce onboarding friction
• Validate environment setup

---

2. Train an RL Policy in Simulation

Next step should introduce basic RL training in simulation.

Suggested configuration:

• Robot: SO101
• Algorithm: Squint
• Task: Lift Cube
• Simulator: ManiSkill

Reasoning:

• Squint is very fast to train (~15 minutes on RTX 4070).
• The Squint method helps reduce the sim2real gap and the paper reports success with zero shot deployment on real hardware
• so101 + Manisill + lift cube because it's tried and tested with squint in https://github.com/aalmuzairee/squint

Recommended walkthrough:

1. Train squint for so101 + lift cube environment
2. Evaluate trained policy in sim

---

3. Train an RL Model for Deployment on Real Hardware

This stage introduces sim2real deployment on real hardware.

Use the same setup as Step 2 to maintain continuity:

• Robot: SO101
• Algorithm: Squint
• Task: Lift Cube
• Simulator: ManiSkill

Primary focus should be on the sim2real gap challenges.

Topics to cover:

Camera Calibration

Provide tooling and walkthroughs for calibration.

Relevant tools:

• Manual camera calibration https://github.com/aalmuzairee/squint/blob/master/deploy_utils/tune_camera.py
• Would be good to consider EasyHEC / SimplyEasyHEC https://github.com/StoneT2000/simple-easyhec

Background Masking

• Include instructions on creating and configuring the background mask to help close the sim2real gap

My Experience with Squint and Sim2Real

The Squint paper reports zero-shot sim-to-real success, I have trained squint and deployed it on my so101, this is what I found:

• Model training was fast and the general process was quite smooth aside from camera calibration
• My camera alignment was imperfect and this is still quite challenging
• I did not get the robot to complete the task successfully but the motion was not bad and I feel like I got a lot further quicker and with better results than with PPO

---

4. HIL RL Training on SO101

Since Squint is SAC.based it should be possible to extend it for HIL (Human-in-the-Loop) RL training.

If so this would make for a natural progression from Step 3.

Topics to cover:

1. Setting up the reward classifier
2. HIL training

---

5. RL Training for VLA Models

I need to get more familiar with RL for VLA models so not going to say much on it atm but it seems like it would make sense as the next step in the progression

[lvjonok]

Hello there!

I am very happy that RL stack will be modernized throughout LeRobot. I would personally benefit a lot from this RL stack not being too fine-tuned for the SO101 robot. Though it is the one people have access the most, but I believe many labs nowadays use some kind of a wrapper around LeRobot to use with their own robots like Franka, UR and so on. Thus, having LeRobot be agile to embodiment (or at least try not to use something outside of the base Robot class) will simplify the integration for the whole community.