Proposal: EXT_WASM_script extension

[JMLX42]

Proposal: EXT_WASM_script

A Vendor-Agnostic WebAssembly-Based Scripting Extension for glTF

Overview

This proposal introduces a new extension, EXT_WASM_script, that enables attaching scripts written in any language compiled to WebAssembly (WASM) directly to glTF assets. By defining a common WASM Component interface for glTF, implementers can embed Turing-complete behaviors in a vendor-neutral, platform-agnostic manner.

This work draws inspiration from KHR_interactivity but opts to leverage existing WASM tooling instead of defining a new node-based, vendor-agnostic programming system from scratch.

Accessibility

1. Language Choice: By targeting WASM, developers can author interactive scripts in their favorite language (C, C++, Rust, AssemblyScript, etc.), removing potential language-barrier or specialized skill-set hurdles.
2. Widespread Platform Support: WASM is a W3C standard supported across platforms (browsers, servers, embedded devices), so the same script can be deployed everywhere glTF is used, helping ensure broad accessibility.

Ubiquity

1. Standardized Runtime: The ubiquitous presence of WASM runtimes (in browsers, game engines, server frameworks, etc.) means minimal friction for glTF viewers to support this extension.
2. Modular Composition: The emerging WASM Component Model standardizes module composition, further easing adoption by allowing interoperable building blocks across multiple platforms and runtimes.

---

Existing Work

KHR_interactivity

The KHR_interactivity proposal introduces a node-based approach to embedding programmatic logic in glTF. Although it is a powerful idea (effectively making glTF Turing-complete), it requires a substantial effort from vendors to map or export existing node graphs (e.g., Unreal Engine Blueprints) to the proposed glTF format.

WebAssembly (WASM)

WASM is a binary instruction format standardized by the W3C for a stack-based virtual machine. It is designed as a portable compilation target, enabling widespread adoption on the web and beyond. Projects like the Bytecode Alliance have championed WASM on multiple platforms and are developing the WASM Component Model for composable modules with platform-agnostic interfaces.

---

Problem to Solve

Interactivity or scripting in a 3D format often relies on platform-specific extensions or custom engines. While KHR_interactivity aims to unify this with a node-based system, exporting and translating existing codebases (especially large ones in various programming languages) into such a new node graph format can be a significant burden.

Key needs include:

• A widely supported binary execution environment that can be used by all glTF viewers
• The ability to write interactive logic in a wide range of programming languages
• A standard interface for hooking user inputs (e.g., keyboard, pointer) and runtime events (e.g., per-frame updates) into the logic controlling the 3D scene

---

Proposal

Introducing EXT_WASM_script

This extension allows glTF files to reference WASM modules that implement a standardized interface for interactivity. A possible example interface, using WIT (the WebAssembly Interface Type syntax), could look like:

Example: WIT Interface for glTF Interactivity

// A hypothetical WIT package for glTF Interactivity
package gltf_interactivity: "0.1.0"

// The 'world' describes how the host and the component connect.
world gltf_interactivity {
    // The host calls these exported APIs to pass data/events to the Wasm component.
    export gltf_interactivity-inputs: gltf_interactivity_inputs
}

// An interface for providing input data and events to the Wasm component.
interface gltf_interactivity_inputs {
    /// Dispatch a pointer event (mouse/touch).
    /// For example: move, down, up, click, etc.
    pointer_event: func(event: PointerEvent)

    /// Dispatch a keyboard event (key press or release).
    keyboard_event: func(event: KeyboardEvent)

    /// Let the component know a frame/tick is happening (time-based update).
    /// Typically called once per frame, with delta-time in seconds.
    tick: func(delta_time: float32)
}

// ----- Records and Enums for Events -----

// Example pointer event data
record PointerEvent {
    event_type: PointerEventType,
    x: float32,     // canvas or viewport x
    y: float32,     // canvas or viewport y
}

// Pointer event "type" enumeration
enum PointerEventType {
    Move,
    Down,
    Up,
    Click,
    // Additional events as needed...
}

// Example keyboard event data
record KeyboardEvent {
    key_code: u32,  // e.g., ASCII/UTF-8 code, or a platform-specific scan code
    is_down: bool,  // true if pressed, false if released
}

Under the hood:

1. Authoring

   • Write interactive logic in any WASM-capable language (C, Rust, AssemblyScript, etc.).
   • Export it to a WASM module that implements the chosen interface (e.g., for handling pointer events, keyboard events, etc.).
   • Reference the WASM module from the glTF using the EXT_WASM_script extension.

2. Runtime

   • A glTF viewer loads the WASM module in a WASM VM.
   • It validates the module’s compliance with the expected interactivity interface.
   • It connects user input events (e.g., pointer or keyboard) and frame updates to the WASM module’s exported functions.

---

Benefits

1. Easy Integration of Existing Code
   Many libraries (physics engines, AI modules, domain-specific logic) already compile to WASM. This extension lets developers drop in widely used solutions without rewriting them for a custom node system.

2. Vendor Neutral
   The specification focuses on hooking into a standardized WASM interface. There is no mandated vendor toolchain, meaning broad adoption can be more frictionless.

3. Broad Language Support
   Authors can choose any language with WASM support, which already includes C/C++, Rust, Go, AssemblyScript, and more. This fosters inclusivity for both seasoned and new developers.

4. Performance & Security
   WASM is sandboxed, providing security benefits in embedded systems or untrusted code environments. It also executes with near-native performance in many scenarios.

5. Cross-Platform Ubiquity
   Virtually every major platform (desktop, mobile, web, VR, AR, IoT devices, servers) can execute WASM modules, making interactive glTF content consistently deployable across diverse targets.

---

Future Enhancements

• WASM Component Model Adoption
  As the Component Model matures, additional interfaces (beyond input events and ticking) can be standardized (e.g., physics, audio, networking), making glTF even more interactive and extensible.

• Shared Memory / Bindings
  Further standardization of communication channels or shared data structures between the glTF scene graph and WASM modules (such as scene node transforms, materials, etc.) could deepen integration.

• Extended Event Types
  Beyond basic pointer/keyboard events, future expansions might include VR/AR controllers, advanced haptics, or platform-specific input devices.

• Built-In Editor Tooling
  Tools like Blender, Unity, or Unreal Engine might eventually provide direct exporters that embed or bundle WASM scripts, streamlining the authoring pipeline for interactive glTF experiences.

---

Conclusion

EXT_WASM_script proposes leveraging the ubiquity and flexibility of WebAssembly to deliver accessible, high-performance scripting in glTF. By aligning with the WASM standard and enabling multiple programming languages, we reduce adoption barriers and simplify the integration of existing codebases. This approach offers a complementary path to the node-based KHR_interactivity, granting developers an additional, readily available tool to bring dynamic, interactive content to glTF.

We welcome feedback from the Khronos community on how best to formalize and refine this extension.

Thank you for reading and considering this proposal!