[Feature]: How to add camera(s) to any morph

[Pinnh]

What feature or enhancement are you proposing?

The api which function is adding camera(s) to any morph, there is only method add_camera of the scene

Motivation

There is no document which describle the method to add cameras to the morphs(box、sphere、drone...) that like the Isaac Sim which add cameras to the morphs by prim paths. The camera is a basic component in drones and vehicles, the autopilot system is heavily rely on the first person view of the visual data (RGB\Infrared)

Potential Benefit

It's obvious

What is the expected outcome of the implementation work?

• [ ]

Additional information

reference:
https://docs.isaacsim.omniverse.nvidia.com/latest/sensors/isaacsim_sensors_camera.html#

[YilingQiao]

Here is an example

Genesis/examples/drone/interactive_drone.py

Line 147 in 0d29e49

scene.viewer.follow_entity(drone)

More details can be found in PR #611

[Pinnh]

If there are more than 2 cameras in a morph, how to code, or if there are two morphs both own one camera? @YilingQiao
The scene's camera follow a entity is not a good idea, the camera is more likely a part of morph , not scene.

[jgleyze]

hello do you think the pr from #618 could help you ? I did the #611 and it was more designed to follow an agent to monitor its training not really something to attach to you robot

[Pinnh]

I want to attach cameras to the robot, we have different purposes. I propose this issue is aim to mention the problem.

[jgleyze]

That's why I am poiting you to #618 which does exactly that :)

[NicolasKuske]

Hi @jgleyze,

That does indeed sound like the solution for @Pinnh and for me as well, as I want to attach a camera to a robot arm.

Would it be possible to give an example how to use the mount function (for the last joint of the Franka panda arm)?

[jgleyze]

Hey @NicolasKuske

Here a quick example, i used this, it attaches a camera to the end effector of the panda and show the camera render in the GUI :

import genesis as gs
import numpy as np

def main():
    
    gs.init()
    
    scene = gs.Scene(show_viewer=True)
    scene.add_entity(gs.morphs.Plane())
    franka = scene.add_entity(gs.morphs.MJCF(file='xml/franka_emika_panda/panda.xml'))
    
    attached_cam = scene.add_camera(GUI=True)
    
    T = np.eye(4)    
    
    T[:3, 3] = np.array([0.1, 0.0, 0.1])
    
    scene.build()
    
    attached_cam.attach(franka.get_link("hand"), T)
    
    for _ in range(1000):
        scene.step()
        attached_cam.render()
        
if __name__ == "__main__":
    main()

You have to tweak Twhich is the offset from the attached link, hope it helps :)

[NicolasKuske]

That's perfect @jgleyze !

I'll try out some stuff with T. Currently it's structure is rather opaque to me. Maybe you can say one or two more words about it? :)

Thank you a lot already, that's quite important for our project!

[duburcqa]

T is a so-called "homogeneous transform". It describes the relative pose of the camera wrt link it is attached to. It gathers the rotation matrix T[:3, :3] and the translation vector T[3; :3].

[jgleyze]

No problem, happy to help :)

Here is a quick introduction to T the homogenous transform matrix : link, especially Fig 7. In simple term it is a matrix that fully encapsulate the position and rotation of a body, we use it so that you don't have to have one vector for the position and one for the rotation.
You can build it easily using functions from geom.py where you can use trans_quat_to_T that takes a position and a quaternion and return T

[NicolasKuske]

For completeness, this is an example configuration for the T-matrix of the attached cam, to look out of the hand link, forward into the world:

T = np.eye(4)    

# Define Rx 180° rotation matrix - rotates around the x-axis of the hand link in the direction away from the robot
Rx_180 = np.array([
    [1,  0,  0],
    [0, -1,  0],
    [0,  0, -1]
])

T[:3, :3] = Rx_180

T[:3, 3] = np.array([0.0, 0.0, 0.08]) # each link has a different wire frame which can be observed in the scene view window (press l)
# for the hand link the z axis goes out upward, and the x axis orthogonal to arm direction outward (foward in x direction)