[FEATURE] Introduce interactive viewer plugins.

[Milotrince]

Description

nice-to-have feature, allows users to integrate interactive viewer plugins.

• Remove dependency on pynput
• gs.vis.keybindings.Keybind callback registration
• in gs.options.ViewerOptions:
  • disable_keyboard_shortcuts replaced with disable_default_keybinds
  • add disable_help_text

Related Issue

Resolves #713

Motivation and Context

• ViewerPlugin enables directly interfacing with pyviewer mouse/keyboard controls.
• Replaces pynput
• Simple keybind callback registration

How Has This Been / Can This Be Tested?

added tests/test_viewer.py

Screenshots (if appropriate):

registered keys automatically show in help text :)

Checklist:

• I read the CONTRIBUTING document.
• I followed the Submitting Code Changes section of CONTRIBUTING document.
• I tagged the title correctly (including BUG FIX/FEATURE/MISC/BREAKING)
• I updated the documentation accordingly or no change is needed.
• I tested my changes and added instructions on how to test it for reviewers.
• I have added tests to cover my changes.
• All new and existing tests passed.

[duburcqa]

This feature looks very convenient!

[Milotrince]

• Should we keep under genesis/ext/pyrender/interaction/plugins/ or genesis/vis/plugins ?
• Do we want to use Vec3, Quat, Color, Ray, etc. classes in the codebase?
• now that pyrender viewer mouse/keyboard interaction is exposed, we can get rid of pynput in examples

[duburcqa]

Should we keep under genesis/ext/pyrender/interaction/plugins/ or genesis/vis/plugins ?

Do you consider this as a "universal" plugin or something pyrender-specific? This would give you the answer :)

Do we want to use Vec3, Quat, Color, Ray, etc. classes in the codebase?

I'm not a fan of these objects. Overly complicated and not very pythonic in my view.

now that pyrender viewer mouse/keyboard interaction is exposed, we can get rid of pynput in examples

That would be great! But we need a way to expose this in a clean way.

[duburcqa]

@Milotrince If you could finish this PR I would be happy to review it and merge it :)

[Milotrince]

@duburcqa Could I get your opinion on the state of the PR? there are some design decisions I'd like your input on!

• removed pynput, see examples/* changes for new usage
• I kept the mouse spring interaction implementation and moved it as a plugin
• there existed a register key system in pyrender viewer but was not exposed in genesis viewer. I made a Keybind/Keybindings class and exposed register_keybind to viewer

I still need to add tests.

[duburcqa]

Could I get your opinion on the state of the PR? there are some design decisions I'd like your input on!

Sure ! I will do it this week ! Could you fix the conflicts in the meantime? Thank you :)

[duburcqa]

Small comment on snapshots PR: https://huggingface.co/datasets/Genesis-Intelligence/snapshots/discussions/6

[duburcqa]

Could you split the introduction of ViewerPlugin feature, with the ability to interactively add markers on geometries from the viewer? Because this PR is getting quite bug already and both features are quite orthogonal. It would make my life easier for the review.

You can open a (stacked) PR for the second part right away, merging on top of this branch instead of main (with draft status).

[Milotrince]

updated snapshot pr: https://huggingface.co/datasets/Genesis-Intelligence/snapshots/discussions/7
posting the "stacked pr" here again (raycaster changes moved here) Milotrince#3

[YilingQiao]

Look forward to landing this PR. I created a GUI on top of that

imgui.mp4

[Milotrince]

@duburcqa some tests are failing from importlib.metadata.PackageNotFoundError: No package metadata was found for gstaichi do i need to worry about this?

[duburcqa]

@duburcqa some tests are failing from importlib.metadata.PackageNotFoundError: No package metadata was found for gstaichi do i need to worry about this?

This is concerning. We should fix this before merging. Weird.

EDIT: the only failure I could find on the CI is this one:

EDIT2: Ok I found the bug!

[github-actions]

⚠️ Abnormal Benchmark Result Detected ➡️ Report

[realmtai]

I'm trying to from genesis.vis.keybindings import Key, KeyAction, Keybind

and getting ModuleNotFoundError: No module named 'genesis.vis.keybindings'

• This is due to gensis-world 0.3.13 does not have the keybindings yet? and will most likely make it into a later release

[duburcqa]

This is due to gensis-world 0.3.13 does not have the keybindings yet?

Yes, you need to install Genesis main branch. It will be part of 0.3.14 release.