Add script to convert LeRobot v3 datasets to unifolm-wma format#43
Closed
brownwa wants to merge 1 commit into
Closed
Add script to convert LeRobot v3 datasets to unifolm-wma format#43brownwa wants to merge 1 commit into
brownwa wants to merge 1 commit into
Conversation
The G1_WBT_Brainco_Pickup_Pillow dataset (and others in the unitreerobotics
HuggingFace collection) are published in LeRobot v3.0 format, but WMAData
expects a different layout:
- {dataset_name}.csv (one row per episode)
- videos/{dataset_name}/{episode_idx}.mp4 (one file per episode)
- transitions/{dataset_name}/{episode_idx}.h5 (state/action per episode)
- transitions/{dataset_name}/meta_data/stats.safetensors
scripts/prepare_lerobot_dataset.py reads the LeRobot parquet/video/stats
files and produces the required layout, including ffmpeg-based per-episode
video clip extraction from the concatenated multi-episode video files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced Apr 4, 2026
Author
Author
|
Superseded by PR #47. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
G1_WBT_Brainco_Pickup_Pillow) are published in LeRobot v3.0 format, butWMAData(insrc/unifolm_wma/data/wma_data.py) expects a different layout — causing aFileNotFoundErroron{dataset_name}.csvat inference time.scripts/prepare_lerobot_dataset.py— a one-time conversion utility that reads the LeRobot parquet/video/stats files and produces the layoutWMADatarequires.DataModuleFromConfig.setup()constructs insrc/unifolm_wma/utils/data.py.What the script generates
Usage
python scripts/prepare_lerobot_dataset.py \ --dataset_dir ~/.cache/huggingface/hub/datasets--unitreerobotics--G1_WBT_Brainco_Pickup_Pillow/snapshots/<hash> \ --dataset_name G1_WBT_Brainco_Pickup_PillowOptional flags:
--camera_key,--action_key,--state_key,--instruction.Test plan
prepare_lerobot_dataset.pyonG1_WBT_Brainco_Pickup_PillowWMADataloads 300 samples without errorreal_eval_server.pystarts and serves successfullyRelated
Fixes the
FileNotFoundErrorreported in issue #XX.🤖 Generated with Claude Code