[WIP] Feat/multithreading

[raldone01]

This is a rewrite of the internals of Metaworld.

It aims to cleanup, improve performance and enable multithreading.

It contains some breaking changes but I think these would be very beneficial to MetaWorld.

Changes/Fixes

• Major __init__ cleanup
• Multithreading support
• Better performance during gym env creation
• No more pickle
• No global state
• Renamed variables to be more descriptive
• Removed broken and confusing Benchmark class
• Breaks existing seeds but
  • Makes the seeding process clearer.
  • Every env has a defined and easy to get seed even when using a benchmark.
• Makes reset actually work with seeds. This will help new users and avoid wasted time because reset just ignores the seed.
• Properly passes through max_episode_steps down to the base env.
• Removed goal-hidden and goal-obserable variants as these can be trivially created by the user by providing the goal_observable kwarg to any benchmark env.
• Removed a lot of magic use of internal environment variables.
• Greatly improve test suit performance and actually cover important cases.
• Testsuite rewrite

Bugs found

• Expert policies fail for reward function v1 on some environments because success reporting is broken. (verified on master)
  • hand-insert-v3
  • peg-insert-side-v3
  • stick-push-v3
• Observation spaces were overriding each other. There would be only one winner. Now a vector with different observation spaces per env type is returned.

Changes TODO

• Update docs
• Update scripts
• Rework evaluation

Questions

• Should the MLX benchmarks only have 40 tasks per env? Or also 50 like the MTX benchmarks?

Out of scope

• Improve the interaction between the base env and all the 50 child envs.
  Ideally avoid inheritance and use composition here.

Includes #569.

[reginald-mclean]

Hi @raldone01,
Thank you for the interesting work you're doing here. We generally don't accept PRs that make large changes across multiple areas of the codebase in one PR. They would be accepted in a PR by PR basis.

There is also a bit of redundancy in the commits you are making compared to our current codebase. For example, we explicitly already support multiprocessing through the use of Gymnasium Vector Environments.

Some of this work is great, such as finding:

• Expert policies fail for reward function v1 on some environments because success reporting is broken. (verified on master)
• Observation spaces were overriding each other

You probably need to provide a justification past "makes more sense" to get any PRs accepted.

Does that make sense?

[raldone01]

Sure. I will finish this and then we could evaluate if anything would be interesting to get upstream. I will use the fork for my projects. I found a lot of the inner workings to be very confusing and performance was a big pain point for me.

Multithreading is key for me since I want to avoid the subprocess overhead. This improves the use case with pytorch for me. The env creation times were also very very slow especially the random vec generation.

Without the lock file this pr cuts out quite a bit of code like -1-2k lines without loosing any functionality.

I don't think I will be able to spend the time to upstream all changes. But if you ever did a version 4 or 5 it would be great if you could take the time to evaluate the state of the fork. Some breaking changes could be avoided.

Also just give this fork a try. The performance difference is really crazy. Especially benchmark creation and teardown.

I can squash/cleanup the git history if that would be important too at some point.