Notes from the Lab -June & July 2025

[ChrisShank]

A packed month of explorations, we wrote a paper, DOM sync, Desktop Interlay, DOM interactions and instruments, HTML pipes, HTML attributes for syntax highlighting, LSPs, and reordering lists, and much more!

June was a quieter month as Chris was traveling and Orion was finishing up some work obligations. We started some compelling explorations but there wasn't enough time to get them where we wanted, so we decided to hold off on a June update. Now that Orion is fully independent researcher, we have so many exciting explorations to share!

Paper

We wrote and submitted a paper, called Live Programming in Hostile Territory, to the LIVE 2025 workshop. This was the first academic paper that both of us have written and it's also the first time we have rigorously attempted to articulate what this project is about. The paper focuses on live programming, as that's the venue that we are submitting to, but we hope the ideas we present extrapolate to adjacent topics like malleable and emancipatory computing. Here's the abstract:

Live programming research gravitates towards the creation of isolated environments whose success is measured by domination: achieving adoption by displacing rather than integrating with existing tools and practices. To counter this tendency, we advocate that live programming research broaden its purview from the creation of new environments to the augmenting of existing ones and, through a selection of prototypes, explore three adversarial strategies for introducing programmatic capabilities into existing environments which are unfriendly or antagonistic to modification. We discuss how these strategies might promote more pluralistic futures and avoid aggregation into siloed platforms.

We also created a Zotero containing a bunch of papers, essays, and videos we referenced (or might reference in the future). There's lots of great resources in there, definitely check it out!

DOM syncing

Orion rewrote the DOM sync attribute from the ground up, improving reliability and performance. We're still exploring what representation we should use in the CRDT and are likely to go with modified version of HAST. This sets us up for representing attributes that contain structured data, such as representing class as an array and style as an object, which means we can have more granular change tracking and updates. The representation that we land on is important because DOM sync backed by a CRDT means that non-web systems can also participate, more on that in the future.

A fun example that we have is this game of chess. The chess board is defined as custom HTML elements, so if you plop the folk-sync attribute on the board you can start playing with other people. The chess board intentionally doesn't encode the rules of chess into itself, that is left to layers above to enforce. Updating the board is as easy as updating a single HTML attribute. We even show how to take a HTML spreadsheet and event propagator to live program a make-shift move log that also syncs.

chess.mp4

Desktop Interlay

The interlay project is aimed at exploring how we can re-appropriate accessibility tree that operating systems make desktop applications adversarial more malleable. It's a WebSocket server that connects web interfaces to accessibility and windowing APIs. We encourage you to read "Adversarial Extension" section of our paper for more context.

axtree.mp4

DOM Interactions and Instruments

A large piece missing from our work stems around the idea of interactions and instruments. Here's a rough sketch:

Some examples of modal instruments are the ones that you might see in the toolbar of a white-boarding application, panning the space, erasing, creating new shapes, etc. But there is a whole slew of instruments that have never existed, imagine an instrument that lets you select part of any web page and author it as markdown, or an instrument that augments any list with the ability to reorder, filter/search, and group it.

What we're interested in is a more general vision of instruments that operate on the DOM and that you can bring around with you as browse the web. It's crucial that interactions are able to be reused and composed together into instruments and that there is a low overhead of producing news types of interactions and instruments.

We currently model interactions as async functions that take in a AbortSignal to cancel them. They compose just like normal functions. Some interactions, like selecting elements, might return a Promise with a selected element. Others might return an empty promise anything because they are performing side effects on the DOM.

We are currently building out a collection of core interactions around selection, creation, brushing, that can be composed together into comprehensive interactions and instruments.

Video frame plucking element

We created a interaction for plucking video frames out of <video> elements. Unlike screenshots, these images are situated in the same environment as the video, so we want to further explore annotating those images with additional context like timestamps linking back to that frame in the video. There is something lurking here about better approaches to being able to annotate videos.

photo-booth.mov

White-boarding App

We've been building a small, multiplayer white-boarding app to prove out and polish up our spatial canvas primitives and start bringing them together. It's all authored in HTML meaning it's easy to add you're own instruments, shapes, etc.

canvas.mov

QR Streaming

As we've been exploring the feasibility of doing WebRTC handshakes over unusual but readily available transports, QR codes and the ability to stream data through them have been captivating. Orion wrote a string codec for compacting data payloads and different protocols for transmitting data through streaming QR codes. Fountain codes are particularly compelling to share data among groups of people because of their ability for many devices can scan the codes at once, and it not mattering when each device starts scanning, only that they collect enough encoded chunks.

qr.mp4

We know it works well because our friend, Chee, was able to transmit all of Moby Dick using it!

HTML pipes/Computational Notebooks

Here we explore what primitives for computational notebooks between DOM elements could look like. There is no extra coordination between DOM elements besides the <folk-pipe> element.

html-pipes.mov

Camera Attribute

The custom HTML attribute turns any <video> element and turn it into a local camera feed. Inspired by Grant Custer's camera collager, we implemented a photo booth, with similar functionality, in a couple of lines of HTML and JS.

pluck.mp4

Syntax highlighting Attribute

Here is a custom HTML attribute that gives any DOM element syntax highlighting without modifying existing text by using the CSS custom Highlighting API. Under the hood we're using PrismJS, so any language it supports can be used.

LSP Attribute

The LSP attribute now supports the TypeScript language server (which also works with plain, old JavaScript), means it supports HTML, CSS, JSON, and JS/TS by default, but provides the ability to register LSPs for any other language. Diagnostics now look better and we also added support for autocomplete! There is still much to do, but the ability to augment plaintext textareas with LSP functionality and no hassle of setting up code editors like codemirror or monaco editor is looking really compelling.

Reorderable Attribute

The reorderable attribute make the children of the element it's applied to interactively reorderable. When available we also use the moveBefore (if possible) to keep the running state of elements as they are reordered. In the future, we'd like to have more list utilities for sorting and filtering and also providing instruments for adding this functionality.

reorder.mp4

Clicklets

Clicklets are a bookmarklet generator that let's you extract structured data from the DOM to use elsewhere. Bookmarklets offer a lightweight way to augment web pages without needing to install web extensions and we're interested in seeing how they can be useful in other explorations.

clicklet.mp4

Thanks for reading, as always we would love to see you sponsor our work if you find it compelling!