Skip to content

[Meshcat] Set object from JavaScript code #22683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

siddancha
Copy link
Contributor

@siddancha siddancha commented Feb 28, 2025

Description

This PR implements:

Meshcat::SetObjectFromThreeJsCode(std::string_view path, std::string_view three_js_lambda)

that lets the user invoke Meshcat's set_object_from_code command (meshcat-dev/meshcat#188) by sending a string of native JavaScript/THREE.js code. The code string contains a JavaScript lambda function that creates a THREE.Object3D instance; this object is then set at path. An example command is:

meshcat.SetObjectFromThreeJsCode(
    "/meshcat/my_object",
    R"""(() => {
        const geometry = new THREE.PlaneGeometry(...);
        const material = new THREE.MeshPhongMaterial(...);
        return new THREE.Mesh(geometry, material);
    })"""
);

Motivation

Currently, the only way to set objects in Meshcat is via SetObject() calls that support a specific set of object types (like Shape, PointCloud etc.). By letting the user create objects via native JavaScript/THREE.js code, Drake's Meschat API can support all THREE.js object types.

Relation to #20941

#20941 proposes something like void SetObjectRaw(std::string_view path, std::string_view threejs_object_json) i.e. passing a THREE.Object3D's serialized JSON. In my view the following are the pros/cons of sending THREE.js code instead.

Pros

  1. Not all THREE.js can be serialized. For example, I want to use the Reflector object [1, 2] to create a reflective floor, as shown in this demo. However, Reflector cannot be serialized because it uses ShaderMaterial [3], which in turn uses dynamic textures that are computed at runtime and cannot be serialized. When I try to serialize by calling Reflector.toJSON(), I get the following warning:
    THREE.Texture: Unable to serialize Texture.
    By letting the user create objects via native JavaScript/THREE.js code, Meschat can support all THREE.js object types, whether they can be serialized or not.
  2. It is often easier to write THREE.js code to create objects than its serialized JSON. The serialization of object types are not well-document, and I have often needed to write native THREE.js code anyway and call the .toJSON() function to understand what the serialization format looks like.

Cons

  1. Specifying the JSON is more efficient if the object's JSON involves large TypedArrays (e.g. Float32Array). For example, setting a PointCloud usually requires specifying a large positions array. Msgpack can efficiently pack float arrays. On the other hand, embedding the floating values into a string might make the string unnecessarily bulky.

In conclusion, this PR might not_resolve #20941, but it can handle most uses cases well. For example, the proposed use case of setting text in Meshcat (#20884) is easily handled by this approach, as demonstrated in the test (see below).

Test example

This PR also adds an example to //geometry:meshcat_manual_test that creates a floating text mesh using Meshcat::SetObjectFromThreeJsCode, as desired by #20884. On running

bazel run //geometry:meshcat_manual_test

we see

image

Resolves #20884


This change is Reviewable

@siddancha siddancha marked this pull request as ready for review February 28, 2025 05:52
@jwnimmer-tri
Copy link
Collaborator

I'm going to mark this +(status: do not review) until meshcat-dev/meshcat#188 makes progress first. In general, yes Drake should plumb through the full set of commands that Meshcat.js offers, so the only question here is really should Meshcat.js offer that new command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Meshcat support for set_object with object.textures.type=_text
2 participants