Bridging the Gap: Rapid Sim-to-Real Transfer via Adaptive Curriculum and Test-Time Calibration

Author: Federico Carollo (s339645)
Institution: Politecnico di Torino
Course: Reinforcement Learning

📖 Abstract

Transferring policies trained in simulation to the real world remains a critical bottleneck due to the "reality gap". This project proposes a unified framework combining Contextual Reinforcement Learning with Test-Time System Identification.

1. Phase 1 (Foundation Model): We train a context-aware policy using DORAEMON (Domain Randomization via Entropy MaximizatiON), an adaptive curriculum that progressively expands the training distribution based on the agent's competence.
2. Phase 2 (Test-Time Calibration): Upon deployment, we perform explicit System Identification using the Cross-Entropy Method (CEM) to estimate latent physical parameters (mass, friction, gravity) from a short batch of observations, without requiring any gradient updates to the policy.

Experiments on Hopper-v4 and Walker2d-v4 demonstrate that this approach allows agents to "survive" and perform efficiently in hostile, out-of-distribution environments where standard robust baselines fail.

---

🏗 Architecture

The project is structured into two main decoupled modules:

1. Adaptive Training (DORAEMON)

Instead of standard Domain Randomization (uniform distribution), we model physical parameters $\xi$ (e.g., torso mass, friction) as Beta Distributions.

• Curriculum: The distribution parameters $(\alpha, \beta)$ are updated via constrained optimization to maximize entropy (diversity) while maintaining a minimum success rate.
• Stability: We implement a "Smart Entropy Thermostat" and lower-bound constraints on $\alpha, \beta$ to prevent catastrophic forgetting and "U-shaped" distributions.

2. Gradient-Free Calibration (CEM)

At test time, the real physical parameters are unknown ("Secret").

• The agent collects a short trajectory ($H \approx 150$ steps).
• CEM Optimizer: Iteratively samples candidate physics parameters, runs parallel simulations ("mental replay"), and refines the estimate by minimizing the MSE between observed and simulated state transitions.
• Context Injection: The estimated parameters $\hat{\xi}$ are fed into the policy $\pi(a|s, \hat{\xi})$.

---

📂 File Structure

File	Description
doraemon_train_v4.py	Main training loop. Manages Parallel Envs, PPO training, and the DORAEMON curriculum updates.
calibration.py	Implements the Cross-Entropy Method (CEM) for explicit System Identification.
doraemon_env.py	Custom Gym Wrapper. Augments the observation space with normalized physical context vectors.
doraemon_optim.py	Contains the constrained optimization logic (scipy.optimize) to update Beta distributions.
full_test.py	Runs a comprehensive evaluation comparing Calibrated vs. Non-Calibrated performance across multiple seeds.
simple_calibration.py	Script to run a single calibration test on a specific environment configuration.
simple_test.py	Quick inference script to visualize the agent's behavior.
pad_wrapper.py	Handles observation padding for compatibility between real environments (no context) and policy inputs.

---

🚀 Installation & Usage

Prerequisites

pip install gymnasium[mujoco] stable-baselines3 numpy scipy pandas

1. Training the Foundation Model

To train the contextual policy with the DORAEMON adaptive curriculum. The script uses SubprocVecEnv for parallel data collection to stabilize gradient estimates.

# Train on Walker2d
python doraemon_train_v4.py --env_id Walker2d-v4 --run_name walker_foundation

# Train on Hopper
python doraemon_train_v4.py --env_id Hopper-v4 --run_name hopper_foundation

2. Running Test-Time Calibration

To test the agent on a specific "Secret" environment configuration (simulating the Real World). This script runs the CEM optimizer to recover hidden parameters from interaction data. Estimated params estimated_params.npy is created.

python simple_calibration.py --env_id Walker2d-v4 --run_name walker_foundation --checkpoint ckpt_latest

3. Full Benchmark Evaluation

To reproduce the comparative results (Calibrated vs. Baseline) across multiple random seeds and generate the CSV report.

python full_test.py --env_id Hopper-v4 --run_name hopper_foundation --checkpoint ckpt_best

4. Quick Inference (Visual Test)

To visualize the agent behavior in real-time with specific parameters. You need to have estimated_params.npy with estimated params during calibration step.

python simple_test.py --env_id Walker2d-v4 --run_name walker_foundation