The pinned dataset revision
lerobot/metaworld_mt50@a59f742d218c903328164257ecf180f9b18018a1
contains all 50 MetaWorld task IDs, but task_id=37 (push-back-v3) and
task_id=38 (push-v3) both map to task_index=40 and the same instruction:
Push the puck to a goal.
This is a quantified follow-up to #1578, originally reported by
@chenkang455 and later auto-closed as stale. The data is not missing a task:
it has 50 IDs and 50 episodes per ID. The defect is a many-to-one semantic
label mapping.
Audit and reproducible artifacts:
https://github.com/sawhney17/metaworld-mt50-task-audit/releases/tag/v0.1.0
Exact scope
| task |
task_id |
task_index |
episodes |
frames |
| push-back-v3 |
37 |
40 |
50 |
8,888 |
| push-v3 |
38 |
40 |
50 |
3,521 |
The collided group is 100 episodes / 12,409 frames. The proposed stable-index
repair preserves push-v3's existing label and relabels only task_id=37:
50 episodes / 8,888 frames.
Semantic-reference consistency check
MetaWorld 3.0.0 at revision
73c6feed5eda4c5269088e914692e8f295c374e2 is used as a pinned semantic
reference, not as a claimed exact generator version. In that reference,
push-back-v3 starts the object at y=0.80–0.85 and its goal at y=0.60–0.70,
while push-v3 starts the object at y=0.60–0.70 and its goal at y=0.80–0.90.
They also use distinct environment and expert-policy classes.
A targeted projection of the pinned dataset state is consistent with those
opposite directions under the reference layout:
- ID 37: initial object y 0.800908–0.849374; goal y 0.600900–0.698324.
- ID 38: initial object y 0.602174–0.698782; goal y 0.800310–0.899855.
Historical provenance and later public reproduction mechanism
The exact generator revision is not established. The dataset's first commit
is dated 2025-02-19; the pinned LeRobot MetaWorld integration PR merged on
2025-10-14. That later public snapshot cannot be claimed as the historical
generator. It does reproduce a mechanism consistent with the published
collision: its config gives the two environment names the same description
while retaining IDs 37 and 38, and its task registry is keyed by description
text.
Independently, the collision already exists in the original v2.0 metadata.
The v2→v3 converter preserves the 49-row registry; it did not introduce the
collision.
Full-corpus validation
I projected six task/index scalar columns from all 492 pinned data Parquets,
covering 204,806 rows and 2,500 episodes:
- 50 task IDs and 49 task indices
- zero null projected rows
- zero duplicate dataset indices
- zero duplicate
(episode_index, frame_index) pairs
- zero raw-vs-episode-metadata mismatches
- exactly one task-ID/task-index collision
Ordered projection SHA-256:
8e1eaa733f6da1f5c5c9685dc9e20a9d9bb56005a2e06bc0e7239244e344877b.
A separate reference-consistency pass projected
observation.environment_state from the exact 32 relevant Parquets. It did
not select image columns.
Proposed repair
- Give
push-back-v3 a distinct, maintainer-approved description, such as
Push the puck back toward the robot to the goal.
- Append that description as
task_index=49; do not shift indices 41–48.
- Change index 40→49 only for
task_id=37: exactly 8,888 rows and 50
episodes.
- Update the task registry, episode metadata,
total_tasks, aggregate stats,
and dataset card.
- Add a config uniqueness regression test.
The audit ships a hash-gated correction specification and non-mutating
sample-level preview. The preview returns index 49 for in-scope rows; it does
not append the registry row, modify Hub data, or constitute a repaired
dataset.
Can maintainers confirm the preferred instruction wording and whether
appending index 49 is the desired compatibility strategy?
Boundaries
This does not claim missing trajectories, raw trajectory corruption, or a
measured policy-score regression. Pipelines consuming task_id, or policies
inferring direction visually, may still distinguish the tasks. The standard
LeRobot language task field cannot.
The pinned dataset revision
lerobot/metaworld_mt50@a59f742d218c903328164257ecf180f9b18018a1contains all 50 MetaWorld task IDs, but
task_id=37(push-back-v3) andtask_id=38(push-v3) both map totask_index=40and the same instruction:Push the puck to a goal.This is a quantified follow-up to #1578, originally reported by
@chenkang455 and later auto-closed as stale. The data is not missing a task:
it has 50 IDs and 50 episodes per ID. The defect is a many-to-one semantic
label mapping.
Audit and reproducible artifacts:
https://github.com/sawhney17/metaworld-mt50-task-audit/releases/tag/v0.1.0
Exact scope
The collided group is 100 episodes / 12,409 frames. The proposed stable-index
repair preserves
push-v3's existing label and relabels onlytask_id=37:50 episodes / 8,888 frames.
Semantic-reference consistency check
MetaWorld 3.0.0 at revision
73c6feed5eda4c5269088e914692e8f295c374e2is used as a pinned semanticreference, not as a claimed exact generator version. In that reference,
push-back-v3starts the object at y=0.80–0.85 and its goal at y=0.60–0.70,while
push-v3starts the object at y=0.60–0.70 and its goal at y=0.80–0.90.They also use distinct environment and expert-policy classes.
A targeted projection of the pinned dataset state is consistent with those
opposite directions under the reference layout:
Historical provenance and later public reproduction mechanism
The exact generator revision is not established. The dataset's first commit
is dated 2025-02-19; the pinned LeRobot MetaWorld integration PR merged on
2025-10-14. That later public snapshot cannot be claimed as the historical
generator. It does reproduce a mechanism consistent with the published
collision: its config gives the two environment names the same description
while retaining IDs 37 and 38, and its task registry is keyed by description
text.
Independently, the collision already exists in the original v2.0 metadata.
The v2→v3 converter preserves the 49-row registry; it did not introduce the
collision.
Full-corpus validation
I projected six task/index scalar columns from all 492 pinned data Parquets,
covering 204,806 rows and 2,500 episodes:
(episode_index, frame_index)pairsOrdered projection SHA-256:
8e1eaa733f6da1f5c5c9685dc9e20a9d9bb56005a2e06bc0e7239244e344877b.A separate reference-consistency pass projected
observation.environment_statefrom the exact 32 relevant Parquets. It didnot select image columns.
Proposed repair
push-back-v3a distinct, maintainer-approved description, such asPush the puck back toward the robot to the goal.task_index=49; do not shift indices 41–48.task_id=37: exactly 8,888 rows and 50episodes.
total_tasks, aggregate stats,and dataset card.
The audit ships a hash-gated correction specification and non-mutating
sample-level preview. The preview returns index 49 for in-scope rows; it does
not append the registry row, modify Hub data, or constitute a repaired
dataset.
Can maintainers confirm the preferred instruction wording and whether
appending index 49 is the desired compatibility strategy?
Boundaries
This does not claim missing trajectories, raw trajectory corruption, or a
measured policy-score regression. Pipelines consuming
task_id, or policiesinferring direction visually, may still distinguish the tasks. The standard
LeRobot language
taskfield cannot.