Aligning results and Questions about latest pretrained models

[YushenZuo]

Hello, thank you for releasing openpi-comet code and the pretrained model checkpoints — they are very helpful.

I am currently reproducing the BEHAVIOR-1K results and would like to report my findings and ask a few questions.

Experimental setup

• Pulled the latest code from the repository (in Jan 07, 2026)
• Aligned configs with the reported setup:
  1. Receding horizon
  2. max_len = 32
  3. RGBWrapper
  4. Corresponding pretrained model
  5. fine_grained_level = 0

1. Reproducing results with task-specific pretrained models

I tested two tasks to align results with previously reported numbers:

• turning_on_radio
• hanging_pictures

Pretrained models

• pi05-turning_on_radio-sft
  https://huggingface.co/sunshk/comet_submission/blob/main/pi05-turning_on_radio-sft.zip

• pi05-hanging_pictures-rft
  https://huggingface.co/sunshk/comet_submission/blob/main/pi05-hanging_pictures-rft.zip

Results

Task	Pretrained model	Avg. Q score
turning_on_radio	pi05-turning_on_radio-sft	1.0 (10 / 10)
hanging_pictures	pi05-hanging_pictures-rft	0.2 (2 / 10)

Compared with the results reported in #3 :

• turning_on_radio matches the reported performance (1.0 vs. 10)
• hanging_pictures is slightly lower (0.2 vs. 3)

2. Testing the latest pretrained models

I also evaluated the latest uploaded models ([Jan 03, 2026]) from: https://huggingface.co/sunshk/openpi_comet
on three tasks:

• turning_on_radio
• hanging_pictures
• picking_up_trash

Models tested

• pi05-b1kpt12-cs32
  https://huggingface.co/sunshk/openpi_comet/tree/main/pi05-b1kpt12-cs32

• pi05-b1kpt50-cs32
  https://huggingface.co/sunshk/openpi_comet/tree/main/pi05-b1kpt50-cs32

Results

Task	Pretrained model	Avg. Q score
turning_on_radio	pi05-b1kpt12-cs32	0.4
turning_on_radio	pi05-b1kpt50-cs32	0.1
hanging_pictures	pi05-b1kpt12-cs32	0.0
hanging_pictures	pi05-b1kpt50-cs32	0.0
picking_up_trash	pi05-b1kpt12-cs32	0.7667
picking_up_trash	pi05-b1kpt50-cs32	0.6333

---

Questions

1. Reproduction gap
   The reproduced score for the hanging_pictures task using pi05-hanging_pictures-rft is slightly lower (0.2). Is this within expected variance, or could there be additional evaluation details/configs I might be missing?

2. Zero score on hanging pictures with latest models
   When using the latest pretrained models (pi05-b1kpt12-cs32, pi05-b1kpt50-cs32), I consistently get 0.0 Q score on the hanging_pictures task.
   Is this expected behavior for these checkpoints, or might there be a mismatch between model and evaluation details/configs?

3. Alignment with your evaluation results
   For the latest pretrained models, could you confirm whether the Avg. Q scores I obtained on turning_on_radio, hanging_pictures, and picking_up_trash tasks are generally consistent with your evaluation results?

4. pt12 vs. pt50 performance trend
   From the results on three tasks, it seems that under the latest models, pi05-b1kpt12-cs32 performs slightly better than pi05-b1kpt50-cs32 (e.g., on turning_on_radio and picking_up_trash tasks). Could you confirm whether this trend is also observed in your internal evaluations on these tasks?

Thanks again for the great work and for making the models public.
Looking forward to your insights.

[hyn1426536274]

Hello, thank you for releasing openpi-comet code and the pretrained model checkpoints — they are very helpful.

I am currently reproducing the BEHAVIOR-1K results and would like to report my findings and ask a few questions.

Experimental setup

• Pulled the latest code from the repository (in Jan 07, 2026)

• Aligned configs with the reported setup:

  1. Receding horizon
  2. max_len = 32
  3. RGBWrapper
  4. Corresponding pretrained model
  5. fine_grained_level = 0

1. Reproducing results with task-specific pretrained models

I tested two tasks to align results with previously reported numbers:

• turning_on_radio
• hanging_pictures

Pretrained models

• pi05-turning_on_radio-sft
  https://huggingface.co/sunshk/comet_submission/blob/main/pi05-turning_on_radio-sft.zip
• pi05-hanging_pictures-rft
  https://huggingface.co/sunshk/comet_submission/blob/main/pi05-hanging_pictures-rft.zip

Results

Task Pretrained model Avg. Q score
turning_on_radio pi05-turning_on_radio-sft 1.0 (10 / 10)
hanging_pictures pi05-hanging_pictures-rft 0.2 (2 / 10)
Compared with the results reported in #3 :

• turning_on_radio matches the reported performance (1.0 vs. 10)
• hanging_pictures is slightly lower (0.2 vs. 3)

2. Testing the latest pretrained models

I also evaluated the latest uploaded models ([Jan 03, 2026]) from: https://huggingface.co/sunshk/openpi_comet on three tasks:

• turning_on_radio
• hanging_pictures
• picking_up_trash

Models tested

• pi05-b1kpt12-cs32
  https://huggingface.co/sunshk/openpi_comet/tree/main/pi05-b1kpt12-cs32
• pi05-b1kpt50-cs32
  https://huggingface.co/sunshk/openpi_comet/tree/main/pi05-b1kpt50-cs32

Results

Task Pretrained model Avg. Q score
turning_on_radio pi05-b1kpt12-cs32 0.4
turning_on_radio pi05-b1kpt50-cs32 0.1
hanging_pictures pi05-b1kpt12-cs32 0.0
hanging_pictures pi05-b1kpt50-cs32 0.0
picking_up_trash pi05-b1kpt12-cs32 0.7667
picking_up_trash pi05-b1kpt50-cs32 0.6333

Questions

1. Reproduction gap
   The reproduced score for the hanging_pictures task using pi05-hanging_pictures-rft is slightly lower (0.2). Is this within expected variance, or could there be additional evaluation details/configs I might be missing?
2. Zero score on hanging pictures with latest models
   When using the latest pretrained models (pi05-b1kpt12-cs32, pi05-b1kpt50-cs32), I consistently get 0.0 Q score on the hanging_pictures task.
   Is this expected behavior for these checkpoints, or might there be a mismatch between model and evaluation details/configs?
3. Alignment with your evaluation results
   For the latest pretrained models, could you confirm whether the Avg. Q scores I obtained on turning_on_radio, hanging_pictures, and picking_up_trash tasks are generally consistent with your evaluation results?
4. pt12 vs. pt50 performance trend
   From the results on three tasks, it seems that under the latest models, pi05-b1kpt12-cs32 performs slightly better than pi05-b1kpt50-cs32 (e.g., on turning_on_radio and picking_up_trash tasks). Could you confirm whether this trend is also observed in your internal evaluations on these tasks?

Thanks again for the great work and for making the models public. Looking forward to your insights.

Hello , I tried the model pi05-turning_on_radio-sft, but the score is almost 0, with only one success or complete failure. Here is my evaluation command:

source .venv/bin/activate
uv run scripts/serve_b1k.py \
  --task_name=turning_on_radio \
  --control_mode=receeding_horizon \
  --max_len=32 \
  policy:checkpoint \
  --policy.config=pi05_b1k-turning_on_radio \
  --policy.dir=<model_path>/pi05-turning_on_radio-sft

conda activate behavior
python OmniGibson/omnigibson/learning/eval.py \
  policy=websocket \
  task.name=turning_on_radio \
  log_path=<log_path> \
  # env_wrapper._target_=omnigibson.learning.wrappers.RGBWrapper

Have you ever tried not using RGBWrapper? Is the impact of this item that significant?