Mixing HTML with 3D components?

[Salketer]

Hello,

I am in the process of migrating from pure threejs + angular app to using this threejs abstraction layer for angular and this is awesome so far.

We currently have 2 major headaches tough:

First one is to be able to introduce HTML directly inside 3D components. Our goal, which we were achieving before migration, is to be able to put (mostly) texts element alongside a mesh. We had all the mechanics of finding the absolute position where it should be displayed according to camera and mesh position. Doing it now with ngx-three would just be a very breeze to use once implemented. The problem is it seems that any template html produced inside a ThCanvas is evaporating so we cannot have our labels displayed.
We are currently passing template refs to a service which then spits it on screen and synchronizes with the view to position them. An example use this feature would be the Html component from @react-three/drei

The second problem is for Picture-In-Picture and is dependent on one. We had made a mecanic to be able to use only one rendering context/canvas to display different scenes or views. ngx-three works very nicely with multiple views and we expect we could make a component leverage this plus the ability to add HTML to create PIP inside html elements using a second canvas where a part of the "main" is copied to.

Anyways, just to be sure, I am not requesting that those features be added in the core of the module as I clearly understand that its goal is to simply wrap threejs into angular components. But we are a bit out of ideas on how to use this great package while be able to expend onto it.

Would you be aware of techniques that we could use to get to our goals without any changes to the package source?
If not, would you consider a PR that allowed for template html's to "bubble up" from the canvas?

[Salketer]

Actually, I've checked out the up-to-date code and the new feature for debug in devMode makes work of the ng-content which is exactly what I needed...

I simply moved it out of the canvas element or else the browser would not add them to the DOM. Now I only need to find an efficient way to keep all those DOM nodes without hurting performance.

[demike]

@Salketer can you take a look at this example: https://demike.github.io/ngx-three/html-example
It displays an iframe with another 3d scene inside. And of course can use any html element inside.

This is actually a port of the r3f / drei's html element.
One of the next steps will be to support the CSS3dRenderer and CSS3DObject and CSS3DSprite:
https://threejs.org/examples/#css3d_periodictable

[Salketer]

Thanks for your quick reply.

I've seen the HTML example, it could work for point 1 but I think I did not understand drei's implementation well enough and tought it was different.

I may have not well explained what we are trying to achieve either... There is actually 2 different use cases, one is for simple UI elements like labels. They should always be visible (not hidden other object in front of them) and should never rotate/resize no matter how near/far they are. This looks to be accomplished in the threejs examples using CSS2DObjects (https://threejs.org/examples/?q=css#css2d_label). Those objects are basically what we had done ourselves by attaching them to a 3DObject and synchronising their positions using the camera projection.

The other thing is more about GUI. We think that using pure HTML/CSS is going to be easier (faster??) than using 3D rendering power. Events and the like will also be easier to handle (unless mistaken, we did not do much tests with the CSS3DObjects or anything). The main reason we'd like to integrate this part inside the ThCanvas is to be able to do the PIPs seemlessly: We'd have a component that would wrap an html canvas and its ThView to render it inside the "master" canvas at the place the HTML canvas is and then the part would be copied to the "child" canvas's 2D context. The main benefit is to be able to re-use most of the 3D context, and we could possibly even use the same scene with just a different camera, like all multi view examples already, but inside HTML elements so they can be bordered by their respective UI etc.

Maybe we have been going the wrong way from the start tough...

[SirukakSosta]

Hello @Salketer.

I can help you implement css2d Labels with this library. I have already build the next components like th-css2drenderer & th-css2dobject. I cant create any pr atm since the code is not good and does not follow @demike 's code rules.

Nevertheless you can achieve what you want by extending this library or by creating a functionality in your angular project.
You will need to access the view.scene.objref & view.camera.objref in order to add your 2dRenderer to the view. And also you will need to include this into the animation frame. I will try to gather the code for you and paste it here.

About the 3d computational power you are refering to, it depends on your specific usecase. If you have lets say 1500 html or more elements then html rendering is not an option. you need to find a way to implement clustering or to render dynamically items that depends on the zoom.

[Salketer]

Thank you @SirukakSosta, since my last posts, we have gone a long way already and we already are achieving what we needed. We are currently in the phase of making it cleaner but it works great.

[demike]

Hello, @Salketer and @SirukakSosta it took longer but now I have added support for multiple renderers.
Now you can use CSS3DRenderer, CSS3DRenderer and WebGLRenderer together.
You can take a look at those examples. Maybe this functionality fits your needs:
https://github.com/demike/ngx-three/blob/main/projects/ngx-three-demo/src/app/multi-renderer-example/multi-renderer-example.component.html

This feature will be in the next release ( next week )