[Evaluation] Full eval is very slow to reproduce — expected? ways to speed up?

[liwq229]

Summary

Thanks for open-sourcing this excellent training/eval stack! I'm reproducing the BEHAVIOR-1K evaluation with the recommended RGBWrapper setting and a finetuned checkpoint, and the end-to-end eval is far slower than I expected. I'd like to confirm whether this is expected and ask for guidance on speeding it up.

Environment

• openpi-comet @ 4bb2aa7, BEHAVIOR-1K @ 684a8305 (StanfordVL)
• Hardware: 8× NVIDIA L20 (46 GB), 384 vCPU, ~1.5 TB RAM; Isaac Sim in Docker
• Checkpoint: finetuned pi05-b1kpt50-cs32, --policy.config=pi05_b1k-base
• Serve: --control_mode=receeding_horizon --max_len=32, server uses XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 (shares the GPU with Isaac Sim)
• Eval: policy=websocket, env_wrapper._target_=omnigibson.learning.wrappers.RGBWrapper (full-res render, per the README recommendation)

How I parallelized

To cover all 50 tasks × 10 instances = 500 episodes, I run 8 workers, one per GPU. Each worker = one policy server + one Isaac Sim eval.py, both pinned to the same L20, and is assigned a fixed slice of tasks (~6–7 each). The README's parallel_evaluator_start_idx/end_idx implementation seems that it does not parallel to 8 gpus.

Observed speed (the actual problem)

Per-episode wall time (n=257 completed):

metric	value
min	19 min
median	~3.9 h (234 min)
mean	~4.5 h (271 min)
p90	7.5 h (450 min)
max	19.4 h

The cost is dominated by long-horizon tasks running to the full timeout. With max_steps: null, eval.py sets max_steps = 2 × average human-demo length, so the per-task cap varies ~10×:

task	cap (sim steps)
canning_food	45,951
storing_food	39,739
loading_the_car	38,453
picking_up_toys	37,782
…	…
turning_on_radio (shortest)	4,300

Questions

1. How long did your official 0.2514 evaluation take, and on what hardware / how many GPUs?

Happy to share my orchestration scripts and logs if useful. Thanks!