No task inputs matching image_primary were found while I am doing the aloha finetuning example

[islexu]

I0206 17:27:57.779100 133821149344896 tokenizers.py:124] No task inputs matching image_primary were found. Replacing with zero padding.

I was wondering does this problem matter in the finetuning process? Or it's a bug and it does not finetune with the "top" image in your aloha sim dataset - features.json.

Thank you in advance!

[endic-sam928281]

Hello, we tried to solve the issue.

This is what we did:

Modify the AlohaGymEnv class to properly include the image_primary observation and ensure it's provided during environment steps.

You can review changes in this commit: endic-sam928281@8a7a6a3.

Caution

Disclaimer: The concept of solution was created by AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

---

Latta AI seeks to solve problems in open source projects as part of its mission to support developers around the world. Learn more about our mission at https://latta.ai/ourmission . If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.

[islexu]

{
"pythonClassName": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
"featuresDict": {
"features": {
"steps": {
"pythonClassName": "tensorflow_datasets.core.features.dataset_feature.Dataset",
"sequence": {
"feature": {
"pythonClassName": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
"featuresDict": {
"features": {
"action": {
"pythonClassName": "tensorflow_datasets.core.features.tensor_feature.Tensor",
"tensor": {
"shape": {
"dimensions": [
"14"
]
},
"dtype": "float32",
"encoding": "none"
},
"description": "Robot action for joints in two arms + grippers."
},
"is_terminal": {
"pythonClassName": "tensorflow_datasets.core.features.scalar.Scalar",
"tensor": {
"shape": {},
"dtype": "bool",
"encoding": "none"
},
"description": "True on last step of the episode if it is a terminal step, True for demos."
},
"is_last": {
"pythonClassName": "tensorflow_datasets.core.features.scalar.Scalar",
"tensor": {
"shape": {},
"dtype": "bool",
"encoding": "none"
},
"description": "True on last step of the episode."
},
"language_instruction": {
"pythonClassName": "tensorflow_datasets.core.features.text_feature.Text",
"text": {},
"description": "Language Instruction."
},
"observation": {
"pythonClassName": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
"featuresDict": {
"features": {
"state": {
"pythonClassName": "tensorflow_datasets.core.features.tensor_feature.Tensor",
"tensor": {
"shape": {
"dimensions": [
"14"
]
},
"dtype": "float32",
"encoding": "none"
},
"description": "Robot joint pos (two arms + grippers)."
},
"top": {
"pythonClassName": "tensorflow_datasets.core.features.image_feature.Image",
"image": {
"shape": {
"dimensions": [
"480",
"640",
"3"
]
},
"dtype": "uint8",
"encodingFormat": "jpeg"
},
"description": "RGB camera observation."
}
}
}
},
"is_first": {
"pythonClassName": "tensorflow_datasets.core.features.scalar.Scalar",
"tensor": {
"shape": {},
"dtype": "bool",
"encoding": "none"
},
"description": "True on first step of the episode."
},
"discount": {
"pythonClassName": "tensorflow_datasets.core.features.scalar.Scalar",
"tensor": {
"shape": {},
"dtype": "float32",
"encoding": "none"
},
"description": "Discount if provided, default to 1."
},
"reward": {
"pythonClassName": "tensorflow_datasets.core.features.scalar.Scalar",
"tensor": {
"shape": {},
"dtype": "float32",
"encoding": "none"
},
"description": "Reward if provided, 1 on final step for demos."
}
}
}
},
"length": "-1"
}
},
"episode_metadata": {
"pythonClassName": "tensorflow_datasets.core.features.features_dict.FeaturesDict",
"featuresDict": {
"features": {
"file_path": {
"pythonClassName": "tensorflow_datasets.core.features.text_feature.Text",
"text": {},
"description": "Path to the original data file."
}
}
}
}
}
}
}

i think the problem might be they are using "top" to define their image_primary?

[nmestvirishvili]

Any luck with this? I'm having the same issue with another dataset