Thoughts on Incrementalization

[viridia]

In my previous post #20821, I tried to narrow the discussion to just reactions, avoiding any discussion of incrementalization. However, since @JMS55 brought up incrementalization, I figured I would write a post that explores this topic.

What is incrementalization?

I first heard the phrase, "Every modern UI framework contains an incrementalization strategy" in one of Raph Levien's talks, although he didn't originate the saying.

In a nutshell, incrementalization refers to a way of dynamically updating a UI hierarchy (or any display hierarchy for that matter) via patching:

• Display hierarchies are trees of nodes
• Trees are patched rather than being replaced wholesale
• Essential state - like text cursor position, selection, IME state, etc. - is preserved between updates.
• Dynamic elements are defined declaratively: we aren't just writing code to manually mutate nodes. (While technically manual mutations are "incremental", they aren't considered a "framework".)
• Dynamic elements are controlled by input parameters; changes happen in response to submitting new parameters.

In effect, incrementalization is a way to implement the following formula: old_tree + new_parameters => new_tree, such that new_tree re-uses the parts of old_tree that didn't change.

A key requirement is that incremental updates should produce results that are indistinguishable from rebuilding everything from scratch: that is, given the same input parameter values, it shouldn't matter whether this is an incremental update or an initial construction. It should look like a "pure function" even when it isn't.

As pointed out in the comment, you can have incrementalization without reactivity: it just means that you have to change the input parameters manually.

Various Strategies

The display tree - what the user actually sees - is the output of an algorithm, however we need more than just the output in order to construct that tree. We need some kind of template or blueprint: a data structure that describes the output we want to generate.

We also need to store some kind of parameter state: the value of the parameters from the previous update, along with any intermediate values derived from them. There might also be local state.

Different UI frameworks approach this problem in different ways. Some of the ones I have seen are:

Long-lived scaffolding that exist separately from the output tree but are connected to it (React, Dioxus). In this approach, the user constructs some alternate tree (the "component hierarchy") that is not the output itself, but rather a scaffolding for building the output. This tree has roughly the same shape as the output.

The user usually doesn't reference the output directly - that is, the user doesn't keep around a reference to the DOM node or entity, instead they hold on to a reference to the scaffolding (an instantiated template), which in turn holds a reference to the output roots. Updating the parameters of this template causes a regeneration of the output.

In this approach, the parameter state is typically stored as properties within the scaffolding nodes.

Temporary scaffolding is a tree that is newly constructed each time (Xylem). In this approach, we regenerate the scaffolding every time from code, throwing it away as soon as we are done. In order for this approach to work, we need three for things to be true:

• The construction of the scaffolding has to be very cheap, it shouldn't involve a lot of memory allocations. There are different ways to do this in Rust, either by using a bump allocator, or by building the tree out of nested tuples.
• The parameter state has to be held external to the scaffolding, in a separate data structure, so that it can be longer-lived.
• The output tree can't radically change shape independently from the scaffolding. The scaffolding expects the shape of the output will match the shape from the previous run, so that it knows where to patch.

Scaffolding embedded within the output as hidden metadata. In the case of Bevy, this can often be done by hidden components (hidden in the sense that they have no effect on the entity's appearance). However, we frequently run into cases where there's no unambiguous single entity that should "own" the scaffolding: examples are templates with multiple roots, or conditional blocks with multiple children. In these cases, we need to insert additional entities, such as ghost nodes, into the tree to hang on to this data.

Differencing

All of the solutions mentioned use some form of "diffing", in the sense that there's some code that compares the old state to the new. However, this may or may not involve actually walking the entire output tree and comparing every property.

In fact, all diffing solutions are selective to some extent: just blindly iterating over all properties (via reflection) doesn't work, because there are some properties that need to be able to update on a different schedule. This is true in React as well: popular animation libraries such as react-spring and framer-motion directly update animatable properties in the DOM, bypassing the diffing mechanism, as this is the only way they can achieve high frame rates cheaply.

This is one of the reasons why React uses a VDOM (Virtual DOM). Not only does it avoid slow DOM operations like appending a node, but the VDOM only contains properties that are meant to be diffed, and doesn't include properties that we want to mutate directly.

For a VDOM, it's pretty simple: if the user mentions a property in the template, then it's meant to be diffed; otherwise leave it alone. This requires a data structure that supports a sparse representation of properties, which entities and components certainly do not.

This has consequences for Bevy: it means that a naive approach which attempts to diff components using reflection won't work, since the diff algorithm would have no way to know which properties are suitable for diffing and which are not (particularly because this decision is highly context-sensitive). Instead, you'd have to build some separate AST-like structure which could then be transformed into entities. This is effectively how Dioxus works, if I understand things correctly.

Some frameworks (like Svelte and Solid) know at compile time which parts of the output hierarchy are static and which are dynamic (they can look for dynamic control-flow elements in the template), and avoid diffing the static parts.

You can take this even further and apply diffing at the parameter state level instead of diffing the output tree, at which point your incremental strategy becomes a kind of memoization of sub-trees rather than a classic "diff". This also avoids wasting time rebuilding parts of the tree that don't change, and which would be discarded after discovering that the diff detected no changes.

Is BSN incremental?

BSN, as currently prototyped, does involve something called a "patch". However, the patch mechanism in BSN is designed to facilitate composition, not dynamic updates. It allows you to bring together multiple sources during construction. It's not designed to modify entities once construction is completed.

Working with BSN macros right now is a little bit like the "temporary scaffolding" approach: you produce an impl Scene, this is used to construct an entity tree, and then the Scene is disposed. It doesn't have any long-lived parameter state, which would require a much different API.

There are a couple of ways that BSN could be extended to support incrementalization:

• Building incremental support into BSN's core design.
• Building incrementalization beneath BSN: using the "embedded" approach to implement incremental updates at the ECS level instead of at the level of scenes and templates.

The second approach is the one I am currently experimenting with. The basic idea is that you have components whose job it is to perform dynamic updates on the tree. As far as BSN knows, these are just components like any other, there's no special support for them. The downside is that, without any kind of special syntactic sugar for dynamism, the syntax for defining these components is a bit boilerplatey. My approach also continues to rely on the ghost nodes feature.

[alice-i-cecile]

Some observations to check my understanding. I don't have much experience with these frameworks, so please grade my answers and correct me!

1. Incrementalization is not used for immediate mode UI frameworks: by definition they just rebuild everything from scratch every frame.
2. The only inherent value of incrementalization is as a performance optimization (which may be very valuable, but there's no other reason to do this).
3. You could have a declarative UI that does not use incrementalization (in fact, this is how immediate mode UI works).
4. Declarative approaches to UI pair well with incrementalization, because they make it easier to understand the complete set of changes / behavior / input.
5. Immediate mode UI similarly needs to concern itself with persisting important state (and this is one of the major sources of complication when building and working with immediate mode UI).
6. When using a "embedded scaffolding" approach, it's better to create extra nodes (in the form of entities) rather than dumping the extra state into a resource etc because then it's easier to integrate with the rest of our tooling: entity cloning, inspection, event bubbling etc.
7. By directly animating the desired properties in the DOM, libraries like react-spring are effectively mutating causal leaf nodes, preventing further reactions.
8. In VDOM-based solutions like React, the entire DOM is a causal dead-end (in terms of incrementalization and reactivity): changes made to the DOM do not trickle down to the rest of the DOM, or back to the VDOM.
9. In a DOM / VDOM solution, only the DOM is actually used for rendering / interaction.
10. In a DOM / VDOM solution, the VDOM is effectively compressed, and stores a smaller, faster-to-work with form of the intended layout.
11. By memo-izing subtrees, you mean that we would compare the final state of subtrees, and then check for differences. In order for this to be effective, we would need some principled rules about how different parts of the UI state could effect other parts: arbitrary global side effects would render this strategy useless.
12. Dioxus works by parsing ASTs in complex ways, and then generating Rust code with the desired behavior by analyzing the actual code provided by the user (rather than sticking to the type level, like Bevy).

[alice-i-cecile]

@JMS55 has corrected me on 2 ("The only inherent value of incrementalization is as a performance optimization"). You also need some sort of strategy to avoid clobbering important state (like events or animations). Simply mass respawning everything each frame will break this (see 5).

bevy_ui currently doesn't have a unified solution for incrementalization, but because it's retained, we're not losing state like this. However, bevy_ui's current design is very hard to scalably make declarative: it requires tons of systems and marker components to update things.

[viridia]

The question that must be asked about immediate mode is, is it truly immediate, or is it just relatively immediate? A truly immediate mode API would be completely stateless, and would leave behind no artifacts other than changed pixels. However, a lot of APIs which advertise themselves as immediate actually do build a hidden, low-level retained data structure. In this, they are not much different than the "transient scaffolding" approach.

For example, in bevy_egui, how does a text widget maintain a persistent cursor position between frames? The user isn't required to manage this, or the selection range, so there must be some kind of hidden state which is "retained". I don't know enough about how this, or bevy's Gizmos, work to say much more.

Mutating DOM properties directly in React: this is the React version of "unsafe": something that is officially sanctioned, but with scary warnings in the documentation. You're free to modify properties as you like, but React will blow away your modifications if it thinks that property is one that it's supposed to manage.

An example of memoizing subtrees would be something like a switch statement: each of the various case clauses in the switch may have its own subtree, but the only thing we need to "diff" is the discriminator: the value that determines which branch we are taking. When that value changes, we despawn and recreate the entire subtree: we don't try to diff between case clauses which may have different shapes.

This brings up the issue of hot reloading, which superficially resembles dynamic switch, in that they both entail replacement of parts of the tree. The difference, though, is that in the dynamic switch case, we can assume that the template source of truth hasn't changed, only the input parameters have; the old version and new version may include different subtrees, but the subtrees that are present have the same shape as before. Whereas in a hot reload scenario all bets are off.

The metaphor I like to use is a complex rail network: dynamic switching is like changing the track switches that route trains to various destinations, whereas hot reloading is like ripping up the track and laying down rails in new locations - the former is a lot more predictable from the framework's perspective.

As a result, hot reloading scenarios tend to be more tolerant of non-incremental replacements: in React for example, touching a source file tends to replace all of the components defined in that source file, as well as any of their descendants; they don't try to do incremental updates within a changed source file.

[alice-i-cecile]

I have some implementation thoughts on this, from an ECS perspective:

In fact, all diffing solutions are selective to some extent: just blindly iterating over all properties (via reflection) doesn't work, because there are some properties that need to be able to update on a different schedule. This is true in React as well: popular animation libraries such as react-spring and framer-motion directly update animatable properties in the DOM, bypassing the diffing mechanism, as this is the only way they can achieve high frame rates cheaply.

This is one of the reasons why React uses a VDOM (Virtual DOM). Not only does it avoid slow DOM operations like appending a node, but the VDOM only contains properties that are meant to be diffed, and doesn't include properties that we want to mutate directly.

For a VDOM, it's pretty simple: if the user mentions a property in the template, then it's meant to be diffed; otherwise leave it alone. This requires a data structure that supports a sparse representation of properties, which entities and components certainly do not.

There are a number of ways we could handle this, if we wanted to :) Ultimately, this is a question of "we need to be able to reason about components, based on the properties of that component type". Some options:

1. Component metadata (possibly stored via components-as-entities), which determine whether or not a specific component should be diffed.
2. Some metadata in the Reflect trait about which fields should be diffed.
3. Trait queries (Add trait queries (by upstreaming bevy-trait-query) #15970), looking for all components that implement a Diff trait.

In most cases, we would want some custom mechanism for "how should this be diffed", to allow for flexibility in terms of which fields we care about for example. The easiest way to do that would be to use the same strategy as FromWorld: provide a blanket implementation of Diff over T: PartialEq, allowing for a nice fallback.

Unsurprisingly, the "component metadata" and "trait query" approaches lead to the same solution here. Trait queries are effectively a convenient conceptual framework to allow for the querying and usage of metadata about each component type.

I think that trait queries are the best approach here (they're more principled than just "more metadata"), and think that using Reflection to do this is likely to be both slow and leak implementation details down into the stack: bevy_reflect should not have to care about these problems, and frankly is the wrong tool for this entirely. Relying on bevy_reflect for production features like this is also bad, because that means we can't disable it (saving huge amounts of binary space) when shipping prod builds.

[alice-i-cecile]

Behavior for "do not diff this component for reactivity on this entity" could be handled fairly easily IMO, by creating opt-out generic components. This would involve a bit of engineering, but should not be treated as a blocker.

[alice-i-cecile]

Trait queries, for the record, rely on "reflection", but not bevy_reflect. In order for them to work, the type must be registered, allowing us to map information about the component type to the traits it has and the function pointers for its methods at runtime. #15030 makes that much less onerous now.

[alice-i-cecile]

My understanding of the problem space is that even if we could do this, we would still need ghost nodes (or something equivalent). There really is just no good spot to store various properties sometimes if we restrict our tree to only "physical" entities.

[nicoburns]

Some frameworks (like Svelte and Solid) know at compile time which parts of the output hierarchy are static and which are dynamic (they can look for dynamic control-flow elements in the template), and avoid diffing the static parts.

Dioxus also does this. It's a key reason for using a macro. The macro inspects the syntax tree and marks anything constant (e.g. string literals - which are pretty common for things like style properties) as not needing to be diffed.

[molikto]

Incremental computation has a very simple definition, not limited to UIs: https://en.wikipedia.org/wiki/Incremental_computing also see https://arxiv.org/pdf/2312.07946#page=5.20

Different people build different incremental solutions for different domains:

• you can build specific incremental algorithm, like incremental graph algorithms (e.g. mentioned in https://arxiv.org/pdf/2312.07946#page=9.53)
• people work on UI build React.
• people work on UI state manager build https://mobx-state-tree.js.org/intro/welcome
• people work on incremental compilation build frameworks like Salsa https://github.com/salsa-rs/salsa
• people work on database build frameworks like Materialize https://materialize.com/docs/get-started/

They are all very different and built only for the problem domain. There are no unified framework that is efficient for all problem domains.

As a very wild example why you should use frameworks purposely build for the probolem domain, someone tried to build a incremental ray tracer by trying to use a the Ocaml library Incremental: https://www.peterstefek.me/incr-ray-tracer . The problem is it can only allow very specific changes of the input. You cannot make changes of the object position. In some sense ReSTIR is a effictive incremental version of ray tracing.

• Building incremental support into BSN's core design.
• Building incrementalization beneath BSN: using the "embedded" approach to implement incremental updates at the ECS level instead of at the level of scenes and templates.

The most traditional way is just don't build incremental/reactivity into BSN/Bevy, and just treat Bevy UI as a retained UI framework, and add a reactive UI layer on top.

If you build ECS incrementality with reactive UI in mind, you ends up only with features that only useful for "reactive UI". But ECS is a general computation framework.

[akriegman]

I'm a big fan of approaches like Svelte and Dioxus that compile down to the minimal necessary computations. I would think that with a simple enough framework the Rust compiler would be able to do that for us though. But I guess Dioxus wouldn't have done it manually if that was the case? I haven't checked but I would think that egui code compiles down about as small as Dioxus code...

So if I can try to simplify here, once per frame we need to know the layout in order to draw the ui. This layout depends on various forms of meaningful state. We can also store various derived state if it helps us compute the layout faster. Because this is Bevy, we're going to store all of our state in the ECS. If we define the layout as a Rust function of the meaningful state, ie a system, then the Rust compiler should give us a fairly optimal conversion from SystemParams to draw calls. The only way we could improve on this is if there was some derived state that was "lower order" than the meaningful state in some sense. And afaict, that's not generally possible. The derived state that these frameworks cache is generally of the same order in size or larger, and so querying the ECS for that derived state to ultimately issue draw calls wouldn't even be cheaper. The reason it's beneficial on the web is because you can only draw through a stateful API.

And also because on the web you only redraw when something changes. The above analysis breaks if we're not drawing every frame. But I guess that's not something we're considering?

Sorry if I have no idea what I'm talking about, it's 1am here and I'm not thinking straight 😅

[akriegman]

But also this argument is assuming that we know what we want the UI to look like as a function of state at compile time. If we want arbitrary dynamic UI, then we'd want an approach that stores a description of the UI in the ECS. Or we can combine the two approaches, let users describe their UI with code most of the time and provide components as an escape hatch.

[tigregalis]

Reactive UI is incrementalisation by construction.

In a reactive system (simplified) you define a signal (something that changes over time, something that you read) and you define an effect (when the signal changes). These are nodes in a graph - signals are sources, effects are sinks. If an effect depends on the value of a signal, that means there is an edge from the signal to the effect.

An analogy for this is a simple circuit: switch, light (and power source).

You can define the relationship between them something like switch -> light - this is a "declarative" or "immediate" view. fn light_state(switch_state: bool) -> bool { switch_state }: Power flows continuously through the circuit, and at any moment if you look at how the power flows through the switch (is it on?) that determines the state of the light.

This is what we want to achieve, and this is what "immediate mode" frameworks achieve "for free" because they just blow away the previous state (effectively).

Returning to the analogy, these elements exist and are continuously operating - they are "live" or "retained" - just like the elements in our UI, or entities and resources in an ECS - to make changes, you have to mutate the world.

Based on the assumption that making changes to the retained world is "expensive", here is where a virtual DOM would come in - you compute a "shadow world", you compare shadows between iterations (diff) and make the minimum changes to the world to synchronise them (patch). Notably a patch expands into many more operations on the actual world and is typically much more costly. Unclear if that's the case in Bevy - how much cheaper could you possibly make the "shadow world" (virtual dom) than the real world?

Returning to the analogy, if you flick the switch (you change the signal), you cut power to the light - this is an "imperative" view. If you change the switch signal to off, the effect is that the light switches off.

Reactivity is "automatic" but "imperative".

Reactivity is a general purpose tool that runs effects when their dependent signals change - that's essentially the sum total of it. It's usually used with the "UI as a function of state" paradigm simply because it's a perfect match: optimally performant, excellent mental model - but it doesn't have to be. An example of this is an effect that tracks a signal but logs the value of that signal - in that case the output is not a function of the state. What typical reactive libraries that target UI do which is what gives them a level of convenience that matches "immediate mode" or "virtual dom" is "dynamic dependency tracking": every time a signal is read while running an effect, you create an edge from the signal to the effect. But explicit tracking is also possible (usually these are exposed as their own tools called watch).

In summary:

1. incrementalisation: something changed, determine what needs to change to uphold the pure input (state) -> output (ui) relationship, then make only those changes.
2. diff and patch: compare two versions of something fake_output1.diff(fake_output2) to determine what needs to change to uphold the pure input (state) -> output (ui) relationship, then make only those changes
3. reactivity: what can change, and what changes need to be made, have all been determined up front. in a UI context, we can use reactivity to uphold the pure input (state) -> output (ui) relationship.

I hope I've convinced you that reactivity is the right paradigm for Bevy: ECS is not immediate mode so we can ignore that, and a virtual DOM / shadow world buys you very little. Reactivity is superior because it's more efficient, but also it assumes statefulness unlike the other two paradigms which have to bolt statefulness on.

In terms of implementation, for Bevy, this means you have to decompose changes into a set of primitives, or general purpose effects:

1. spawn an entity
2. add a component to an entity (an effect)
3. update the value of a component on an entity
4. remove a component from an entity
5. add a child
6. remove a child
7. despawn an entity

Notably, these primitives already exist, so it's just a matter of wrapping them in effects when required.

For example, a desugared (after macro expansion or whatever mechanism is used to construct this, perhaps Templates) counter application would look something like:

let value = commands.create_signal(0);
let target = commands.spawn(Button);
commands.create_effect(|world| {
    let value = *value.get(world);
    world.entity(target).insert(Text(value.to_string());
});
target.observe(|_: On<Pointer<Click>>, world: &mut World| { value.get_mut(world) += 1; } )

The actual code is a little bit more verbose than this, but it works today. I spawn an entity for every signal, and the effects simply wrap pre-registered SystemIds, and I use change detection on the signal data to determine what effects need to run.

One thing to note is that the change can be even more fine-grained and efficient than this, for example, you could just clear and write to the String. The "shape" of the desugared code doesn't change much, just the specificity of the effect. However, I think Component-level may be a good enough starting point.

let value = commands.create_signal(0);
let target = commands.spawn(Button, Text::default());
commands.create_effect(|world| {
    let value = *value.get(world);
    world.entity(target).get_mut::<Text>(target).0.clear();
    write!(world.get_mut::<Text>(target).0, "{value}").ok();
});
target.observe(|_: On<Pointer<Click>>, world: &mut World| { value.get_mut(world) += 1; } )

Having said all this, I think there are probably places in the engine that could benefit from greater incrementalisation but I don't believe that should be conflated or mixed up with providing a good API for users to write UIs declaratively just because the shape of the problem seems similar.

For more on reactivity, this is an excellent summary of how modern reactive libraries that target UI work: https://book.leptos.dev/appendix_reactive_graph.html