Cannot reproduce Figure 10 pick success rate (28%) with public pi05_droid_jointpos checkpoint

[www-Ye]

Summary

I'm trying to reproduce the zero-shot evaluation results in Figure 10 using the public pi05_droid_jointpos checkpoint and the official evaluation code, but consistently get significantly lower success rates on the Pick task (~18% vs. reported 28%).

Setup

• Checkpoint: pi05_droid_jointpos, downloaded via gsutil cp -r gs://openpi-assets/checkpoints/pi05_droid_jointpos . as instructed in the README
• Benchmark: FrankaPickHardBench_20260206_json_benchmark (procthor-objaverse, 2000 episodes)
• Eval config: PiPolicyEvalConfig (policy_dt_ms=66.0)
• task_horizon_steps: 300
• Eval command:

  from molmo_spaces.evaluation.eval_main import run_evaluation
  results = run_evaluation(
      eval_config_cls=PiPolicyEvalConfig,
      benchmark_dir="...FrankaPickHardBench_20260206_json_benchmark",
      checkpoint_path="checkpoints/pi05_droid_jointpos",
      task_horizon_steps=300,
      num_workers=16,
  )

• OpenPI server: serve_policy.py --policy.config=pi05_droid_jointpos

Results

Setting	Pick Success Rate
Paper Figure 10 (π0.5)	28%
My result (objaverse 20260131, horizon=500)	~15.9%
My result (objaverse 20251016_from_20250610, horizon=500)	~18.4%
My result (objaverse 20251016_from_20250610, horizon=300)	~14.1%

Questions

1. Which objaverse version? The code accepts both 20260131 and 20251016_from_20250610. Which was used for the reported results?

2. Any other configuration details not documented (e.g., specific server-side settings, random seeds) that might affect reproducibility?

Environment

• Python 3.10, conda environment
• MuJoCo with EGL rendering
• OpenPI server on the same machine

Any guidance on reproducing the reported numbers would be appreciated.

[omarrayyann]

hello @schmmd! The results for the pick task in the paper uses assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231 with 450 steps. Try:

from molmo_spaces.evaluation.eval_main import run_evaluation
results = run_evaluation(
    eval_config_cls=PiPolicyEvalConfig,
    benchmark_dir="assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231",
    checkpoint_path="checkpoints/pi05_droid_jointpos",
    task_horizon_steps=450,
    num_workers=5,
)

Let me know if you’re unable to reproduce the results with this corrected benchmark and task horizon.

[www-Ye]

hello @schmmd! The results for the pick task in the paper uses assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231 with 450 steps. Try:

from molmo_spaces.evaluation.eval_main import run_evaluation
results = run_evaluation(
eval_config_cls=PiPolicyEvalConfig,
benchmark_dir="assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231",
checkpoint_path="checkpoints/pi05_droid_jointpos",
task_horizon_steps=450,
num_workers=5,
)
Let me know if you’re unable to reproduce the results with this corrected benchmark and task horizon.

Thank you so much for the prompt and helpful reply! Really appreciate it. The pick results now match the paper using FrankaPickDroidMiniBench with task_horizon=450.

Would you mind sharing the correct benchmark paths and task horizons for the other three tasks in Figure 10 (open, close, pick & place) as well? Here's what I'm currently using — just want to make sure these are correct:

Task	Benchmark	Episodes	task_horizon
Pick ✅	procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231	1000	450
Open	ithor/FrankaOpenHardBench/FrankaOpenHardBench_20260206_json_benchmark	2000	450
Close	ithor/FrankaCloseHardBench/FrankaCloseHardBench_20260206_json_benchmark	2000	450
Pick & Place	procthor-10k/FrankaPickandPlaceDroidMiniBench/FrankaPickandPlaceDroidMiniBench_20260111_json_benchmark	1000	450

Also, could I ask about the training data — is the pi05_droid_jointpos checkpoint trained on the official DROID dataset, or is there a separate dataset/link used?

Thank you so much for your time and patience!

[omarrayyann]

Great! We use these benchmarks for each task:

pick: assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231
open: assets/benchmarks/molmospaces-bench-v1/ithor/FrankaOpenDataGenConfig/FrankaOpenDataGenConfig_20260123_json_benchmark
close: assets/benchmarks/molmospaces-bench-v1/ithor/FrankaCloseDataGenConfig/FrankaCloseDataGenConfig_20260123_json_benchmark
pick-place: assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickandPlaceDroidMiniBench/FrankaPickandPlaceDroidMiniBench_20260111_json_benchmark

We did not train these policies ourselves, they're in the official openpi bucket (gs://openpi-assets/checkpoints). As far as I'm aware, it's a pi05_droid checkpoints finetuned on DROID's commanded jointpos values.

[www-Ye]

Great! We use these benchmarks for each task:

pick: assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickDroidMiniBench/FrankaPickDroidMiniBench_json_benchmark_20251231 open: assets/benchmarks/molmospaces-bench-v1/ithor/FrankaOpenDataGenConfig/FrankaOpenDataGenConfig_20260123_json_benchmark close: assets/benchmarks/molmospaces-bench-v1/ithor/FrankaCloseDataGenConfig/FrankaCloseDataGenConfig_20260123_json_benchmark pick-place: assets/benchmarks/molmospaces-bench-v1/procthor-10k/FrankaPickandPlaceDroidMiniBench/FrankaPickandPlaceDroidMiniBench_20260111_json_benchmark

We did not train these policies ourselves, they're in the official openpi bucket (gs://openpi-assets/checkpoints). As far as I'm aware, it's a pi05_droid checkpoints finetuned on DROID's commanded jointpos values.

Thank you so much for sharing the benchmark paths — this is incredibly helpful! I've updated my evaluation settings accordingly.

Just one more quick question: is the task_horizon 450 for all four tasks, or does it vary across tasks?

Thanks again for your patience and all the help!

[BlGene]

@www-Ye
I was the task_horizon 450 for all four tasks.