[Question] How to create a clickable button in Isaac Sim when using XR

[sashansamarajeewa]

Build Info

Describe the versions that you are currently using:

• Isaac Lab Version: 2.3.0
• Isaac Sim Version: 5.1.0

Description

I’m using Isaac Sim with XR and I’d like to create an in-world button that the XR user can press to trigger Python callbacks (e.g. start/stop).

I’ve tried using omni.ui widgets with WidgetComponent and UiContainer,

class SimpleButtonWidget(ui.Widget):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        with ui.ZStack():
            with ui.VStack():
                self._btn = ui.Button("Start", clicked_fn=self._on_click)

    def _on_click(self):
        print("button clicked") 

def create_xr_button_panel():
    def on_constructed(widget_instance):
        pass

    widget_component = WidgetComponent(
        SimpleButtonWidget,
        width=0.3,
        height=0.1,
        resolution_scale=20,
        unit_to_pixel_scale=30,
        update_policy=sc.Widget.UpdatePolicy.ALWAYS,
        construct_callback=on_constructed,
    )

    space_stack = [
        SpatialSource.new_translation_source(Gf.Vec3d(0.3, 0.9, 1.5)),
        SpatialSource.new_rotation_source(Gf.Vec3d(
            math.radians(90), 0.0, 0.0,
        )),
    ]

    ui_container = UiContainer(widget_component, space_stack=space_stack)
    return ui_container

The widget shows up correctly in XR, but the button is not clickable from inside XR.

What is the recommended/supported way to create interactive buttons that can be pressed by the XR user inside Isaac Sim? If there is an example or sample extension that shows a clickable XR button, could you point me to it?

Thank you

[RandomOakForest]

Thank you for posting this. I'll move this to our Discussions section for follow up. @peterd-NV, @rwiltz for vis.

[newbieKD]

A possible workaround is to use keyboard inputs to replace button interactions in the VR interface. This works well as a temporary.

[sashansamarajeewa]

Thank you for the suggestion. My goal is to have buttons controllable by the XR user inside the simulation without relying on keyboard inputs.

[rwiltz]

Hi @sashansamarajeewa if you are using CooudXR we do support adding UI in the headset and wiring it up in Isaac Lab. Take a look at this line here: https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab/isaaclab/devices/openxr/openxr_device.py#L84 this subscribes Isaac Lab to the message bus. At the end of the same file you can see how the messages are handled.

To send the message, you then use the CloudXR SDK to add UI in the client application (using native code IE swift UI on the AVP) and connect the UI to CloudXR. Example: https://github.com/isaac-sim/isaac-xr-teleop-sample-client-apple/blob/main/IsaacXRTeleopClient/TeleopControlView.swift

If you are not using CloudXR, you need enable enable XR UI kit parameter if you desire for those elements rendered by Isaac Lab to be interactive. Let me know if that's the path you are interested in.

[sashansamarajeewa]

Thank you for the reply. Could you please explain how it can be done with XR UI kit in Isaac Lab?