[FEATURE] Camera and viewer able to follow and entity

[jgleyze]

Fixes #538 and #554

Hello !

I made changes to the camera and viewer class so that they can now follow an entity in various ways.

For both the viewer and any camera, once they are created, you can :

scene = gs.Scene(
            sim_options=gs.options.SimOptions(dt=0.02, substeps=2),
            viewer_options=gs.options.ViewerOptions(
                max_FPS=50,
                camera_pos=(2.0, 0.0, 2.5),
                camera_lookat=(0.0, 0.0, 0.5),
                camera_fov=40,
            ),
            show_viewer=True,
        )

entity = scene.add_entity() #load any enity

scene.viewer.follow_entity(entity)

follower_camera = scene.add_camera(res=(640,480),
                                        pos=(0.0, 2.0, 0.5),
                                        lookat=(0.0, 0.0, 0.5),
                                        fov=40,
                                        GUI=True)
 
follower_camera.follow_entity(entity, fixed_axis=(None, None, 0.5), smoothing=0.5, fix_orientation=True)

In this code both the viewer and follower_camera will follow the entity. The viewer will be rigidly linked to the entity, no smoothing (a bit nauseating if you do locomotion ahah!). The second one will have a smoothed motion on the xy plane and a fixed height of 0.5m.

fixed_axis is used to fix the camera movement on a chosen axis [x, y, z], if set to None the camera is able to move on this axis during the following. If set to a float the camera will stay fixed at this value on this axis. This does not fix its orientation. Hence won't change where the camera looks at, just its position the camera can still look up with a fixed z

fix_orientation is used alongside the fixed_axis because even though you fixed some axis the camera will rotate due to the call of lookat, this can lead to unwanted movement. For example, if fixed on the z height in a locomotion case the robot's gait might lead to some z movement and those would be repercuted on the camera orientation doing ups and downs.

Added code

• camera.py

  • Added attributes relative to the following
  • Added function follow_entity to register the followed entity with some arguments related to the cameras' movement
  • Updated the render function to compute the camera's position with update_following

• viewer.py

  • Basicaly the same, variation on the functions to call to set the camera's position

• interactive_drone.py

  • Removed the update_camera function since it is now redundant, the viewer is directly set to follow the drone

• go2_env.py

  • Added a follower camera on the side of the robot, the camera have fixed height and orientation

• go2_train.py

  • Added -v / --vis to enable the gui for the viewer and follower camera

Examples

You can run

python examples/drone/interactive_drone.py
python examples/locomotion/go2_train.py -v

TODO

The camera only follow the first env due to camera.py

I don't really know how to get the id of each env here...

[YilingQiao]

Thanks for your PR. It works great on my side.

[YilingQiao]

I merged this PR, removed the examples on Go2, and kept only the drone example because I do not want to modify the Go2 example. I will also add more documentation about this feature.

[BoruiHe]

Thank you for your effort. I am looking for the same feature for recording robots' action. I followed the instruction above and added a new camera/visualizer that can freely move along all axes for recording. It turns out that the camera never follow the loaded robot.

Here is the code for minimal reproduction. Run GS.py, then you will get a video.mp4 of 4 seconds. Based on what I can tell from the video, the camera never moves. Do I miss any point? Thank you in advance.

[jgleyze]

Thanks for the comment. i found the bug and it's related to the TODO in the end of my first post, your code work if you build multiple env with scene.build(n_envs=N). I might have time to look into it this week, if you have any idea feel free to share!

[BoruiHe]

A temproal fix: use 'scene.build(n_envs=1)' instead of 'scene.build()'.

I think I found the bug.
Found this in /genesis/vis/camera.py line 309~311:
entity_pos = self._followed_entity.get_pos()[0].cpu().numpy() if entity_pos.ndim > 1: entity_pos = entity_pos[0]

When we do not pass the n_envs parameter to scene.build(), the 'get_pos()' returns a tensor that has 1 less dimension. (num_env, x, y, z) -> (x,y,z). Only when n_envs is set to an integer, get_pos()[0] returns the postion in the first environment. Otherwise, it catches the X coordinate. Even in the correct case, entity_pos.ndim still returns 1, which means the branch in line 310 and 311 probably never works.

Here is my idea: use entity_pos = self._followed_entity.get_pos().cpu().numpy(). After this modification, the camera follows my robot on my side, scene.build() and scene.build(n_envs=1) output the same video. Correct me if I was wrong.

Btw, I feel that the camera is not following my robot the way I expercted. I want it to move and look at somewhere like eyes. Setting fix_orientation=True, the camera still looks at the given direction when robot falls. Setting it to False, the camera looks at the floor. I am thinking that an eye-like camera should update its lookat_pos at every step. Hope I can do something about it.

[jgleyze]

thanks for the insight, I'll look into your change and submit it, concerning your second point you might want to look at #618 wich mount a camera to a link, my code is rather used to follow an entity while looking at it to monitor the locomotion of an agent in training for exemple, not to see the world through its eyes.