Make convert_v3_to_v2.py non-destructive by default#719
Open
asierarranz wants to merge 1 commit into
Open
Conversation
The script replaced the source v3.0 dataset in place and deleted existing _v3.0/_v2.1 sibling dirs without warning, silently destroying data. Write the converted dataset to --output (default: sibling _v2.1 dir) and gate in-place replacement behind an explicit --in-place flag. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
If you run
scripts/lerobot_conversion/convert_v3_to_v2.pytoday, it converts your dataset in place and the source v3.0 dataset is gone. Nothing in--helpwarns about it. It happened to us: the script ate a dataset of 40 teleop episodes and we had to regenerate it from the source HDF5 files.Why people run this script: there is a format gap between the two worlds right now. Modern LeRobot (v0.6.0) records everything as v3.0, while the native tooling in this repo (finetune scripts, data loader) still expects v2.1. So anyone recording with LeRobot today and fine-tuning through the native path has to go through this converter, and #677 even points v3.0 users straight to it. That is a lot of first-time users about to run a script that can eat their only copy. This PR makes the default behavior safe.
What changed:
--output(default: a sibling directory with a_v2.1suffix). The source is never touched.shutil.rmtreeon existing siblings without warning.--in-placeflag: it warns first and keeps the original in a sibling_backup_v3.0directory. Mutually exclusive with--output.--in-placeso its downstream paths keep working. Both READMEs and--helpdocument the new behavior.How tested: 7 new unit tests in
tests/scripts/test_convert_v3_to_v2.py(destination resolution, refusal to overwrite source or existing dirs, backup naming, flag exclusion),ruff checkandruff format --checkclean with the repo config, CLI smoke-tested with both flag combinations.Note:
--force-conversionstill removes an existing source snapshot; that behavior predates this PR and is left as is to keep the diff focused.