Notes from the Lab - March 2025

[ChrisShank]

This month, we restructured the repository in preparation for publishing. We extracted and optimized geometry utilities into a separate package, implemented Bounding Volume Hierarchies for efficient spatial queries, and began exploring custom HTML attributes as a compliment to custom elements. We solved matrix math challenges for handling nested space transformations, and worked towards spatial keyboard navigation.

Our peer-to-peer experiments resulted in prototypes that use audio transmissions, QR codes, and various signalling methods for establishing p2p connections. We also created demonstrations for DOM synchronization, web applets, graph-based zoomable user interfaces for hypertext, self-contained HTML module systems, and asynchronous event propagators.

Restructured the repo

Up to this point the repo has been duct taped together using Bun and a Vite app with some path aliases. It gave the feel of a monorepo without the additional tooling required, but we were really just kicking the can down the road. So we finally bite the bullet...

• switched to NodeJS,
• moved over to a actual monorepo using NPM workspaces and wireit
• setup Github actions and started deploying to Github pages.

This also sets us up to start publishing alpha versions to NPM soon. Moving to Node allows us to debug tests with VSCode's Javascript Debug Terminal and use the Deopt Explorer to get a better view into JS performance. While none of this stuff is relevant to the goals of the project it's necessary to make sure this stuff stays out of our way. Orion also scaffolded a new repo to record and compare benchmarks. There's also a CONTRIBUTING.md now!

Geometry utilities

As part of the restructuring, we extracted out our core geometry utilities into a new package.

This included rewriting and optimizing them from a class-based approach to interfaces with utility functions. We noticed better performance and better code splitting. The later is notable because in the long run it allows us to keep adding more utilities while keeping the bundle size as small as possible. Also programming to interfaces improves usability, although we need to be mindful that could come at the cost of JS performance via polymorphism.

We added an interface and utilities for Bounding Volume Hierarchies allowing us to efficiently query spaces with many shapes. Refactoring the canvas primitives and other experiments to use this package is ongoing.

BVH Demo

Screen.Recording.2025-03-30.at.12.10.15.AM.mov

Canvas Primitives

We started some explorations around using custom HTML attributes as an alternative to custom elements. The benefit is we can augment existing DOM tree with spatial canvas behaviors without needing to modify the tree structure, quite useful in adversarial environments! We have very early prototypes of a folk-shape and folk-zoomable attributes. One tough thing about custom HTML attributes (when you aren't the browser) is when those custom attributes need to render things (e.g. shapes need a shape overlay when focused). It's unclear where we will land with using custom attributes or custom elements for these primitives.

We added the ability for shapes to be toggled between being in document-flow (i.e. position: static) and being in space (i.e. position: absolute) if no x and y coordinates are given to a shape. This will be useful in pulling parts of a document apart into the spatial canvas. We spent some time thinking through how spaces and shapes can be composed arbitrarily and why that doesn't currently work.

Document flow demo

Screen.Recording.2025-03-30.at.12.13.48.AM.mov

Keyboard accessibility is important in the context of this project because we want these spatial canvas primitives to feel that they are part of the web platform. Also, broadly speaking keyboard accessibility of infinite canvases is pretty poor. The struggle is that document-flow keyboard focus isn't an effective way to navigate through spatial canvas. Additionally for our shape element it doesn't seem good if we want the handles for each shape to be apart of document flow? And for the shape attribute, what happens when it's element already has a focus behavior? Maybe what we really want is mode for spatial keyboard navigation! Still early on, but the idea is that option/alt + tab toggles between spatial keyboard mode and regular document flow. When it's one you can only navigate between shapes in the canvas using the tab + arrow keys and modify the position/width/height/rotation of those shapes with the keyboard.

Spatial keyboard navigation demo

Screen.Recording.2025-03-30.at.12.23.30.AM.mov

• We also started building out a web extension that allows us to use these spatial canvas primitives in adversarial environments. Questions like:
  • Can we make turn web pages that aren't ours into a spatial canvas?
  • Can we turn other people's web pages into collaborative places via the P2P primitives (talked about below)?
  • Can we enable different forms of connections to be made between web pages?

Canvasify via web extension demo

o.mp4

Spaces

One problem of shapes being in a zoomable space is that the space transforms cause a disconnect between pointer events (screen coordinates) and the actual shape coordinates. But we also wanted to solve this problem not just for the typical pan + zoom space, but arbitrary spaces, potentially even spaces that are nested. Turns out we just needed to wrap our heads around some matrix math!

Space Transforms demo

pointer-o.mp4

To handle the above this means that spaces and shapes need to know about each other. Up to this point they don't because they are designed to work standalone by default. So we added a pretty simple event interface that shapes dispatch when are connected and disconnected.

Composing shapes and spaces

Screen.Recording.2025-03-22.at.00.18.41.mp4

Primitives for graph-based zoomable user interface (ZUI). We showed that it's possible to take a set of markdown files with wiki-style links (e.g. Obsidian, Bear) and transform it into a hypertext ZUI. This graph-based structure where a space can contain itself appears very similar to a manifold.

ZUI demo

zui-o.mp4

Space morph
We have been exploring more semantic morphisms for spaces. More to do here, at some point we will write down why these things are important to this project.

demo

Screen.Recording.2025-03-22.at.02.24.43.mp4

Tools & Instruments

One of the crucial parts of the spatial canvas are tools and instruments that operate on the canvas, the shapes in the canvas, and even the DOM more broadly. We're interested in tools that are part of the canvas itself, not necessary in fixed position overlay. Once tools are in the canvas, then things you can typically do to shapes you can also start doing for tools. For instance take copy and pasting. If you can copy and paste shapes in a canvas, why cant we also copy and paste tools? Why cant I easily add my own tools to the canvas? Why can't I paste in other peoples tools? One approach is to make parts of the canvas self-contained so that if they are copied, not only do you copy the instance of that tools (i.e. it's current configuration) but also it's source code. Shapes and tools are already custom HTML elements so it's easy to serialize them to HTML. If we define a simple protocol where those elements have a property referring to a CDN for their source code then we can also copy that to the clipboard. Now we can paste these tools and shapes into potentially any other web page (assuming a web extension injects a little bit of code to handle that).

Copy and Paste demo

Screen.Recording.2025-03-17.at.00.02.54.mp4

The tools that you usually see in most white boarding applications are modal. Only one in selected at at a time and when they receive focus they capture all pointer events that would typically allow you to interact with the elements in the canvas. So we started exploring what some kind of interface/protocol/base class could look using event capture. A edge case quickly arose where HTML media elements suppress pointer events for security reasons While that put a damper in things for now there are ways to get around that that we're hoping to explore next month.

peer-to-peer and DOM sync

Orion dove deep into peer-to-peer (p2p) WebRTC, in the true local-first sense using ways to directly connect web pages through local signaling like QR code or audio protocols or piggy backing off open networks like BitTorrent trackers or mmqt. The goals is low-tech signalling that requires no infrastructure deployment and can easily connect web pages together. We have POCs for:

Audio transmission using GGWave

demo

H1MAVkeLi0Y6lNTC.mp4

QR code transmission

demo

-GCZRhVrfE418onS.mp4

QR code + audio transmission

demo

ihq3EoFFsEf7qMAJ.mp4

• mmqt signaling
• GunDB signaling + webRTC
• A demo for peer presence

We're getting close to finding some nice interfaces for signaling/network adapters and network transports, and how they interact with syncing mechanisms like CRDTs. We'll eventually put them in their own package. Orion has also been optimizing the signaling formats/payloads, the less data we need to send for signaling, the more reliable and quicker it will be.

Additionally there are some prototypes around syncing DOM trees via an custom HTML attribute. This isn't a novel idea (see Webstrates and PlayHTML), but the flexibility to easily swap in the networking layer surely does, especially in the context of p2p WebRTC. While also being easy to augment existing DOM trees. This should just work with our spatial canvas primitives since they already changes to the DOM.

Misc

• dataglyphs are rich, inline diagrams

demo

Screen.Recording.2025-03-30.at.1.32.01.AM.mov

The DOM is a substrate, right? Since web applets can be embedded using a custom HTML element, it's pretty easy to demonstrate how they can work with our canvas primitives and event propagators.

Web applets demo

Untitled4.mov

• How to make a self-contained HTML file have it's own module system

Asynchronous event propagators

Asynchronous event propagators demo

Screen.Recording.2025-03-22.at.19.56.38.mp4