Summary
I full-fine-tuned LingBot-VLA on a single-task pick (cx001_select_banana) with the vision tower, QwenVL backbone, and action expert all trainable (freeze_vision_encoder=false, train_expert_only=false). Training loss and open-loop MAE look fine, but in closed-loop inference the policy reliably approaches the target object and then stalls — the end-effector hovers in a near-correct pre-grasp pose and the gripper never closes, so the pick is never completed.
Setup
- Base:
LingBot-VLA (75-D), task cx001_select_banana (dual-arm 14-D, 12 pose + 2 grippers).
- Loss: flow-matching,
chunk_size=30, lr=3.5e-5, 50k steps, 8×H100 FSDP2.
- 14-D action zero-padded to
max_action_dim=75 to keep base ckpt's action_in_proj / action_out_proj weights intact.
- Dataset minmax-normalize on; gripper range fixed
[0, 4.5].
- Inference: VR + real robot, ODE steps 4 / 10, ckpts 50k / 100k — same symptom.
Observation
Gripper channel (idx 6 / 13, 0≈closed, ~4.5≈open) collapses to ~0 almost immediately and stops varying in any task-meaningful way, even when the arm is correctly positioned over the object:
[..., -0.013] [..., 0.987] [..., 0.000] [..., 0.019] [..., 0.000] ...
Questions
- For single-task fine-tuning on small data, is full unfreezing (vision + VLM + expert) actually recommended, or should
freeze_vision_encoder=true be the default?
- Is the 14→75 zero-pad expected to be safe for the gripper dim end-to-end? With 61/75 channels permanently 0, could the loss be effectively down-weighting the 2 gripper dims and pushing them toward the dataset mean?
- Any known recipe / mask / loss weighting for the gripper channel to avoid this "approach-but-no-grasp" collapse?
Happy to share configs, loss curves, and a rollout video.
Summary
I full-fine-tuned LingBot-VLA on a single-task pick (
cx001_select_banana) with the vision tower, QwenVL backbone, and action expert all trainable (freeze_vision_encoder=false,train_expert_only=false). Training loss and open-loop MAE look fine, but in closed-loop inference the policy reliably approaches the target object and then stalls — the end-effector hovers in a near-correct pre-grasp pose and the gripper never closes, so the pick is never completed.Setup
LingBot-VLA(75-D), taskcx001_select_banana(dual-arm 14-D, 12 pose + 2 grippers).chunk_size=30,lr=3.5e-5, 50k steps, 8×H100 FSDP2.max_action_dim=75to keep base ckpt'saction_in_proj/action_out_projweights intact.[0, 4.5].Observation
Gripper channel (idx 6 / 13,
0≈closed,~4.5≈open) collapses to ~0 almost immediately and stops varying in any task-meaningful way, even when the arm is correctly positioned over the object:[..., -0.013] [..., 0.987] [..., 0.000] [..., 0.019] [..., 0.000] ...
Questions
freeze_vision_encoder=truebe the default?Happy to share configs, loss curves, and a rollout video.