Moveable components with adjustable Camera

[BullFlog]

Hello,

I am experimenting with react-three-fiber / react-spring use-cannon. I wanted to ask if it is possible to move a component in 3d if I can adjust the camera as well (with OrbitControls from "drei").

If the camera is static I can use:

  api.position.set(
    (state.mouse.x * state.viewport.width) / 2,
    (state.mouse.y * state.viewport.height) / 2,
    0     
  );

Of course, if I now move the camera, everything gets inverted.

Is there a reliable way to implement this?

Thank you