Skip to content

[Dataset] Restore distinct push-back-v3 label in lerobot/metaworld_mt50 (50 task IDs -> 49 labels) #4409

Description

@sawhney17

The pinned dataset revision
lerobot/metaworld_mt50@a59f742d218c903328164257ecf180f9b18018a1
contains all 50 MetaWorld task IDs, but task_id=37 (push-back-v3) and
task_id=38 (push-v3) both map to task_index=40 and the same instruction:
Push the puck to a goal.

This is a quantified follow-up to #1578, originally reported by
@chenkang455 and later auto-closed as stale. The data is not missing a task:
it has 50 IDs and 50 episodes per ID. The defect is a many-to-one semantic
label mapping.

Audit and reproducible artifacts:
https://github.com/sawhney17/metaworld-mt50-task-audit/releases/tag/v0.1.0

Exact scope

task task_id task_index episodes frames
push-back-v3 37 40 50 8,888
push-v3 38 40 50 3,521

The collided group is 100 episodes / 12,409 frames. The proposed stable-index
repair preserves push-v3's existing label and relabels only task_id=37:
50 episodes / 8,888 frames.

Semantic-reference consistency check

MetaWorld 3.0.0 at revision
73c6feed5eda4c5269088e914692e8f295c374e2 is used as a pinned semantic
reference, not as a claimed exact generator version. In that reference,
push-back-v3 starts the object at y=0.80–0.85 and its goal at y=0.60–0.70,
while push-v3 starts the object at y=0.60–0.70 and its goal at y=0.80–0.90.
They also use distinct environment and expert-policy classes.

A targeted projection of the pinned dataset state is consistent with those
opposite directions under the reference layout:

  • ID 37: initial object y 0.800908–0.849374; goal y 0.600900–0.698324.
  • ID 38: initial object y 0.602174–0.698782; goal y 0.800310–0.899855.

Historical provenance and later public reproduction mechanism

The exact generator revision is not established. The dataset's first commit
is dated 2025-02-19; the pinned LeRobot MetaWorld integration PR merged on
2025-10-14. That later public snapshot cannot be claimed as the historical
generator. It does reproduce a mechanism consistent with the published
collision: its config gives the two environment names the same description
while retaining IDs 37 and 38, and its task registry is keyed by description
text.

Independently, the collision already exists in the original v2.0 metadata.
The v2→v3 converter preserves the 49-row registry; it did not introduce the
collision.

Full-corpus validation

I projected six task/index scalar columns from all 492 pinned data Parquets,
covering 204,806 rows and 2,500 episodes:

  • 50 task IDs and 49 task indices
  • zero null projected rows
  • zero duplicate dataset indices
  • zero duplicate (episode_index, frame_index) pairs
  • zero raw-vs-episode-metadata mismatches
  • exactly one task-ID/task-index collision

Ordered projection SHA-256:
8e1eaa733f6da1f5c5c9685dc9e20a9d9bb56005a2e06bc0e7239244e344877b.

A separate reference-consistency pass projected
observation.environment_state from the exact 32 relevant Parquets. It did
not select image columns.

Proposed repair

  1. Give push-back-v3 a distinct, maintainer-approved description, such as
    Push the puck back toward the robot to the goal.
  2. Append that description as task_index=49; do not shift indices 41–48.
  3. Change index 40→49 only for task_id=37: exactly 8,888 rows and 50
    episodes.
  4. Update the task registry, episode metadata, total_tasks, aggregate stats,
    and dataset card.
  5. Add a config uniqueness regression test.

The audit ships a hash-gated correction specification and non-mutating
sample-level preview. The preview returns index 49 for in-scope rows; it does
not append the registry row, modify Hub data, or constitute a repaired
dataset.

Can maintainers confirm the preferred instruction wording and whether
appending index 49 is the desired compatibility strategy?

Boundaries

This does not claim missing trajectories, raw trajectory corruption, or a
measured policy-score regression. Pipelines consuming task_id, or policies
inferring direction visually, may still distinguish the tasks. The standard
LeRobot language task field cannot.

Metadata

Metadata

Assignees

No one assigned

    Labels

    configurationProblems with configuration files or settingsdatasetIssues regarding data inputs, processing, or datasetsevaluationFor issues or PRs related to environment evaluation, and benchmarks.examplesIssues related to the examplespoliciesItems related to robot policiesprocessorIssue related to processortestsProblems with test coverage, failures, or improvements to testing

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions