Problem Description
I followed the official README of openpi-comet to reproduce the 50-task BEHAVIOR-1K results using the provided pi05-b1kpt50-cs32 checkpoint (the one specifically trained for the 50 tasks).
Expected results (according to leaderboard + paper):
Validation success rate ≈ 0.1440 / Q-score ≈ 0.1830 (submission value)
Post-competition optimized Q-score ≈ 0.345
Actual results:
On BEHAVIOR-1K v3.7.0 and v3.7.2 (stable releases): Q-score only ~0.03 (3%) across 50 tasks.
On latest main branch (as of 2026-04-14): evaluation fails with a series of import / NameError / Hydra errors (detailed below).
Reproduction Steps (strictly followed README)
Cloned openpi-comet (latest main) and BEHAVIOR-1K.
Copied all custom files:
cp -r openpi-comet/src/behavior/learning/* BEHAVIOR-1K/OmniGibson/omnigibson/learning/
Started policy server (confirmed checkpoint + norm_stats loaded successfully):
CUDA_VISIBLE_DEVICES=3 python scripts/serve_b1k.py
--task_name=turning_on_radio
--control_mode=receeding_horizon
--max_len=32
policy:checkpoint
--policy.config=pi05_b1k-base
--policy.dir=../pi05-b1kpt50-cs32
Ran evaluation with correct flags:
python OmniGibson/omnigibson/learning/eval.py
policy=websocket
task.name=turning_on_radio
log_path=./logs/turning_on_radio_test
env_wrapper.target=omnigibson.learning.wrappers.RGBWrapper
Errors on latest main branch
Even after commenting out all non-essential imports (lerobot, gello, BaseRobot, register_omegaconf_resolvers, etc.) as suggested in the debugging process, we still hit:
ModuleNotFoundError: No module named 'lerobot.datasets...'
NameError: name 'HF_LEROBOT_HOME' is not defined
ModuleNotFoundError: No module named 'gello.robots.sim_robot.og_teleop_cfg'
ImportError: cannot import name 'BaseRobot' from 'omnigibson.robots'
NameError: name 'BaseRobot' is not defined (in type hints)
hydra.errors.ConfigCompositionException: You must specify 'policy'... (even when policy=websocket is explicitly passed)
NameError: name 'register_omegaconf_resolvers' is not defined
On stable v3.7.x the evaluation runs but success rate stays near zero.
Environment
openpi-comet: latest main
BEHAVIOR-1K: tested v3.7.0, v3.7.2, and latest main
Python: 3.10 (behavior conda env)
Checkpoint: official pi05-b1kpt50-cs32
GPU: RTX 5090 (serve logs show successful JAX/Orbax checkpoint restore)
Request
Could the team please clarify:
Which exact BEHAVIOR-1K commit / tag is officially supported with the current openpi-comet custom files (the 2026-01-06 version)?
Is there a recommended stable commit hash for BEHAVIOR-1K that matches the pi05-b1kpt50-cs32 weights?
Any additional flags or post-processing steps needed to reach the reported Q-score of ~0.345?
Happy to provide full logs, serve output, or test any suggested commit. Thank you!
Problem Description
I followed the official README of openpi-comet to reproduce the 50-task BEHAVIOR-1K results using the provided pi05-b1kpt50-cs32 checkpoint (the one specifically trained for the 50 tasks).
Expected results (according to leaderboard + paper):
Validation success rate ≈ 0.1440 / Q-score ≈ 0.1830 (submission value)
Post-competition optimized Q-score ≈ 0.345
Actual results:
On BEHAVIOR-1K v3.7.0 and v3.7.2 (stable releases): Q-score only ~0.03 (3%) across 50 tasks.
On latest main branch (as of 2026-04-14): evaluation fails with a series of import / NameError / Hydra errors (detailed below).
Reproduction Steps (strictly followed README)
Cloned openpi-comet (latest main) and BEHAVIOR-1K.
Copied all custom files:
cp -r openpi-comet/src/behavior/learning/* BEHAVIOR-1K/OmniGibson/omnigibson/learning/Started policy server (confirmed checkpoint + norm_stats loaded successfully):
Ran evaluation with correct flags:
Errors on latest main branch
Even after commenting out all non-essential imports (lerobot, gello, BaseRobot, register_omegaconf_resolvers, etc.) as suggested in the debugging process, we still hit:
On stable v3.7.x the evaluation runs but success rate stays near zero.
Environment
Request
Could the team please clarify:
Which exact BEHAVIOR-1K commit / tag is officially supported with the current openpi-comet custom files (the 2026-01-06 version)?
Is there a recommended stable commit hash for BEHAVIOR-1K that matches the pi05-b1kpt50-cs32 weights?
Any additional flags or post-processing steps needed to reach the reported Q-score of ~0.345?
Happy to provide full logs, serve output, or test any suggested commit. Thank you!