Poor inference performance in the real robot

[gzyabc]

I have collected 200 robot trajectories and trained them using Lingbot-VLA for a total of 200,000 steps, saving checkpoints every 50,000 steps.

I tested the 100,000-step checkpoint and found the inference performance to be very poor, even though the training loss dropped to around 0.02. I am training on A100 GPUs with a batch size of 32 (16 per node). The dataset is in LeRobot format.

I am struggling to identify the root cause. Could you provide some insights? Is it more likely an issue with the dataset size, action normalization, training configuration, or deployment pipeline

[ccmCCMfk]

My dataset is also in LeRobot format, with 121 samples. I added waist degrees of freedom, making the action dimension 20 in total. The training loss can go down to 0.02, but the inference performance is still very poor. Could this be due to limitations in the model’s generalization ability?

[gzyabc]

@ccmCCMfk Could you share which robot platform you are using? I am working with the Agibot G2. After deploying the checkpoint from 100,000 training steps, my impression is that the model hasn't actually 'learned' the tasks at all.Could you share your inference videos? Since we replaced the G2's end-effector from a gripper to a dexterous hand, I am temporarily unable to share my own videos.

[YangS03]

Hi @gzyabc @gzyabc, did you check the open-loop results (sampling an raw episode from the dataset and running the dataloader as the “robot”) to ensure that the training configuration is correct? The model should be able to fit the trajectory. If the results are incorrect, the issue is likely due to the data, embodiment, or training configuration. If not, the problem is likely in the deployment.

[gzyabc]

@YangS03 I haven't performed an open-loop test yet. To clarify, does the procedure involve feeding three images from the original dataset into the model and comparing the predicted actions against the ground truth labels?"

I believe my dataset is solid because I previously trained a PI model (Physical Intelligence) with it, and the resulting inference actions were essentially normal. However, when using the exact same dataset with Lingbot-VLA, the predicted actions are incorrect. Here is my training configuration，could you help me check if there are any errors or misalignments?

[YangS03]

@gzyabc, can you provide the info.json in lerobot dataset？

[gzyabc]

@YangS03 Here is the info.json file. I am currently training using the CustomizedRobotwinDataset class. Given the current issues, should I be using the LiberoDataset instead? I would appreciate it if you could clarify which dataset class is best suited for this specific data format.
{
"codebase_version": "v2.1",
"robot_type": null,
"total_episodes": 201,
"total_frames": 215941,
"total_tasks": 1,
"total_videos": 603,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:201"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path": "videos/chunk-{episode_chunk:03d}/{video_key}/episode_{episode_index:06d}.mp4",
"features": {
"observation.state": {
"dtype": "float32",
"shape": [
16
]
},
"actions": {
"dtype": "float32",
"shape": [
16
]
},
"observation.images.hand_left": {
"dtype": "video",
"shape": [
480,
640,
3
],
"info": {
"video.height": 480,
"video.width": 640,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 30,
"video.channels": 3,
"has_audio": false
}
},
"observation.images.hand_right": {
"dtype": "video",
"shape": [
480,
640,
3
],
"info": {
"video.height": 480,
"video.width": 640,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 30,
"video.channels": 3,
"has_audio": false
}
},
"observation.images.top_head": {
"dtype": "video",
"shape": [
480,
640,
3
],
"info": {
"video.height": 480,
"video.width": 640,
"video.codec": "av1",
"video.pix_fmt": "yuv420p",
"video.is_depth_map": false,
"video.fps": 30,
"video.channels": 3,
"has_audio": false
}
},
"timestamp": {
"dtype": "float32",
"shape": [
1
],
"names": null
},
"frame_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"episode_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"index": {
"dtype": "int64",
"shape": [
1
],
"names": null
},
"task_index": {
"dtype": "int64",
"shape": [
1
],
"names": null
}
}
}

[YangS03]

@gzyabc Can you provide the raw dataset (or a subset of it)? We will test the training code on our device.

[LuFan31]

Hi @gzyabc, before we proceed with resolving this issue, I’d like to confirm a few details with you:

1. Are you using scripts/compute_norm_robotwin_5.py to compute the normalization statistics for your own data? In this part of the script, it is specified that actions across all dimensions do not subtract the state. This implies that during deployment, you should not add the state back to the predicted actions. While the decision of whether to subtract the state from the action during data normalization and training depends on your specific needs, the key is to maintain strict consistency between training and deployment.

2. Please ensure that the data_type and norm_type parameters are identical when calling the Normalizer during both the training and deployment stages. Also, verify that the input norm_stats are loaded from the same normalization file.

Additionally, I recommend using MSE flow-matching loss for real-robot post-training. You can enable this by simply commenting out train.loss_type: L1_fm in your training configuration.

Once you have verified these points, please share your norm_stats_file with us so we can take a look.

[gzyabc]

@LuFan31 @YangS03 Thanks for your help and for answering all my questions
1.Yes, I previously used your scripts to calculate the norm_stats. It seems these were calculated using those 200 episodes of data.The content of the file is as follows:
{
"norm_stats": {
"actions": {
"mean": [
1.5962085723876953,
-1.161338210105896,
-1.5159064531326294,
-1.721285343170166,
-0.4381822347640991,
0.04852576181292534,
-0.031516481190919876,
5.695648670196533,
-1.4997355937957764,
-1.1641603708267212,
1.463254451751709,
-1.6542468070983887,
0.24998871982097626,
0.0321204736828804,
0.03580693155527115,
0.23197393119335175
],
"std": [
0.24911963939666748,
0.059795018285512924,
0.22557958960533142,
0.15656477212905884,
0.07913357764482498,
0.10345225036144257,
0.08915555477142334,
25.222993850708008,
0.28527119755744934,
0.09248322993516922,
0.28027257323265076,
0.22486664354801178,
0.4236021935939789,
0.13574457168579102,
0.19503654539585114,
1.2247437238693237
],
"q01": [
0.7806676626205444,
-1.3868314027786255,
-1.673017978668213,
-1.853799819946289,
-0.7531708478927612,
-0.049955785274505615,
-0.3775872588157654,
0.0,
-1.7332537174224854,
-1.4264025688171387,
0.42062410712242126,
-1.9753570556640625,
-1.5704280138015747,
-0.3403366506099701,
-0.6858362555503845,
0.0
],
"q99": [
1.7298959493637085,
-0.9497198462486267,
-0.5957045555114746,
-1.2012436389923096,
-0.24510645866394043,
0.3399491310119629,
0.13247132301330566,
119.97599792480469,
-0.6825665235519409,
-0.8363630771636963,
1.91752028465271,
-0.9742109775543213,
0.7154814004898071,
0.33988168835639954,
0.5228894948959351,
0.9961559772491455
],
"q02": [
0.8532505035400391,
-1.3478002548217773,
-1.6408334970474243,
-1.8046534061431885,
-0.7001645565032959,
-0.02347990870475769,
-0.31857848167419434,
0.0,
-1.7332537174224854,
-1.3878090381622314,
0.6637848615646362,
-1.9230470657348633,
-1.5408049821853638,
-0.3400583267211914,
-0.44047796726226807,
0.0
],
"q98": [
1.7298959493637085,
-1.0187469720840454,
-0.7099074125289917,
-1.2740532159805298,
-0.3027220368385315,
0.33981263637542725,
0.07295602560043335,
119.97599792480469,
-0.8351230621337891,
-0.9133581519126892,
1.8763946294784546,
-1.0826493501663208,
0.6019264459609985,
0.33960333466529846,
0.47289347648620605,
0.9961559772491455
]
},
"observation.state": {
"mean": [
1.5962072610855103,
-1.1613378524780273,
-1.5159058570861816,
-1.7212817668914795,
-0.4381829798221588,
0.04852670803666115,
-0.031515538692474365,
34.42148208618164,
-1.49973726272583,
-1.1641576290130615,
1.4632545709609985,
-1.6542459726333618,
0.24999120831489563,
0.03211916610598564,
0.03580956906080246,
21.94243049621582
],
"std": [
0.24911916255950928,
0.0597970113158226,
0.2255801260471344,
0.1565990298986435,
0.07913584262132645,
0.10345499217510223,
0.08915628492832184,
53.654441833496094,
0.2852749526500702,
0.0924961194396019,
0.28029128909111023,
0.22487670183181763,
0.42360126972198486,
0.13575032353401184,
0.1950361579656601,
44.883487701416016
],
"q01": [
0.780945897102356,
-1.3868274688720703,
-1.6726635694503784,
-1.853817105293274,
-0.7532334327697754,
-0.04978552460670471,
-0.377356618642807,
0.0,
-1.7329267263412476,
-1.4262232780456543,
0.42051613330841064,
-1.9754681587219238,
-1.5702524185180664,
-0.34012091159820557,
-0.6845744848251343,
0.0
],
"q99": [
1.7298896312713623,
-0.9499087929725647,
-0.595372200012207,
-1.2012414932250977,
-0.24512726068496704,
0.3399822413921356,
0.13215726613998413,
119.97599792480469,
-0.6821542978286743,
-0.836756706237793,
1.9175344705581665,
-0.9743368625640869,
0.714553713798523,
0.3398892879486084,
0.5232930183410645,
119.97599792480469
],
"q02": [
0.852934718132019,
-1.3477975130081177,
-1.640825867652893,
-1.8046691417694092,
-0.7002227306365967,
-0.023318976163864136,
-0.31860554218292236,
0.0,
-1.7329267263412476,
-1.3880136013031006,
0.6636967658996582,
-1.9227769374847412,
-1.5406080484390259,
-0.34012091159820557,
-0.4396399259567261,
0.0
],
"q98": [
1.7298896312713623,
-1.0183918476104736,
-0.7102648019790649,
-1.273749828338623,
-0.30295705795288086,
0.33984580636024475,
0.07264643907546997,
119.97599792480469,
-0.8350540399551392,
-0.913560152053833,
1.876006007194519,
-1.0831555128097534,
0.6020153760910034,
0.33975040912628174,
0.47245752811431885,
119.97599792480469
]
}
},
"count": 215936
}
2.As far as I remember, I haven't modified the contents of the Normalizer class.I recall that during training, # "action_is_pad": normalized_item["action_is_pad"] was throwing an error. As a workaround, I changed it to: "action_is_pad": torch.zeros(normalized_item["actions"].shape[0], dtype=torch.bool). Could this modification be affecting the action prediction accuracy?I also modified some field names within the CustomizedRobotwinDataset class.For example：
data_type='customized',
norm_type={
"observation.images.top_head": "identity",
"observation.images.hand_left": "identity",
"observation.images.hand_right": "identity",
"observation.state": data_config.norm_type,
"action": data_config.norm_type,
},
3.What is the best way for me to upload the dataset for you?

[YangS03]

Can you use modelscope or huggingface?

[LuFan31]

@gzyabc I notice that the key of action in your data is actions (as shown in your norm_stats file), while you use action when calling CustomizedRobotwinDataset with the mentioned norm_type. So the action does not apply normalization during training and deployment. Please follow this instruction and make sure that all keys in your data match those when calling Normalizer. Moreover, deployment like here is also dependent on Normalizer, we need to keep parameters identical during both the training and deployment stages.

After applying the correct normalization, the values should be normalized around [-1 ,1], so you can also check the values.

[gzyabc]

@YangS03 Here is the link to my LeRobot dataset uploaded on ModelScope. Please let me know if there are any issues with the dataset.Thank you for your help.https://modelscope.cn/datasets/gzyabc/G2_200/files.
@LuFan31 Thanks for the reminder. Upon inspection, I found that the Normalizer class in the inference code is different from the one in CustomizedRobotwinDataset. Could this discrepancy be the root cause of the issue?

[LuFan31]

@YangS03 Here is the link to my LeRobot dataset uploaded on ModelScope. Please let me know if there are any issues with the dataset.Thank you for your help.https://modelscope.cn/datasets/gzyabc/G2_200/files. @LuFan31 Thanks for the reminder. Upon inspection, I found that the Normalizer class in the inference code is different from the one in CustomizedRobotwinDataset. Could this discrepancy be the root cause of the issue?

Yep. Considering the default training config does not apply CustomizedRobotwinDataset, so I have not modified the data_type in Normalizer class in the inference code. Instead, I have reminded here and highlighted the importance of keeping consistent between training and deployment stages.

Please fix these and try again😊