SoA Diff Serializer Emits No Data for Recycled Entity IDs with Unchanged Field Values

[americanjeff]

Note I have a proposed fix for this which I'm cleaning up. I'll send a PR for it soon.

Summary

When an entity is removed and its ID is recycled by a new entity, the SoA diff
serializer may silently skip serializing the new entity's component data if the
new entity's field values happen to equal the old entity's values. The client
receives the new entity via the observer (AddEntity) but its component arrays
remain undefined.

The root cause is that removeEntity does not clear typed-array slots, leaving
stale values at the freed EID. The diff serializer's shadow copy for that slot
still holds the old value. When the recycled EID is serialized again with the
same value, hasChanged() sees shadow === current and emits nothing.

---

Repro

Minimal case

import { createSoASerializer, createSoADeserializer, u16 } from "bitecs/serialization";

const Force = { count: u16([]) };
const clientForce = { count: u16([]) };

const serverSerialize = createSoASerializer([Force], { diff: true });
const clientDeserialize = createSoADeserializer([clientForce], { diff: true });

// Step 1: establish shadow — entity at slot 5 has count=1
Force.count[5] = 1;
const initial = serverSerialize([5]);
clientDeserialize(initial);
// clientForce.count[5] === 1  ✓

// Step 2: entity at slot 5 is removed and a NEW entity is created at slot 5
// (bitECS LIFO recycling), also with count=1
Force.count[5] = 1;  // same value

// Step 3: serialize again — shadow[5]=1, current=1 → no diff emitted
const delta = serverSerialize([5]);
console.log(delta.byteLength);  // 0  ← BUG: new entity received no data

// Client creates new entity at slot 5 (via observer deserializer)
clientForce.count[5] = undefined as unknown as number;
clientDeserialize(delta);
console.log(clientForce.count[5]);  // undefined  ← client state corrupted

With observer

import { addComponent, addEntity, createWorld, removeEntity } from "bitecs";
import {
  createObserverSerializer,
  createSoADeserializer,
  createSoASerializer,
  u16,
} from "bitecs/serialization";

const world = createWorld();
const Networked = {};
const Force = { count: u16([]) };
const clientForce = { count: u16([]) };

const observerSerialize = createObserverSerializer(world, Networked, [Force]);
// BUG: no way to wire removals into the SoA serializer without patching

const serverSerialize = createSoASerializer([Force], { diff: true });
const clientDeserialize = createSoADeserializer([clientForce], { diff: true });

const e1 = addEntity(world);
addComponent(world, e1, Networked);
Force.count[e1] = 1;

observerSerialize();
const initial = serverSerialize([e1]);
clientDeserialize(initial);
// clientForce.count[e1] === 1  ✓

removeEntity(world, e1);
const e2 = addEntity(world);       // bitECS recycles e1's ID
addComponent(world, e2, Networked);
Force.count[e2] = 1;               // same value as old entity

observerSerialize();               // emits RemoveEntity(e1), AddEntity(e2)
const delta = serverSerialize([e2]);
console.log(delta.byteLength);     // 0  ← new entity's data never sent

clientForce.count[e2] = undefined as unknown as number;
clientDeserialize(delta);
console.log(clientForce.count[e2]); // undefined  ← client state corrupted

---

Root Cause

When removeEntity is called, the EID is pushed onto the free-list and the
array slot is not cleared but the SoA serializer doesn't know it. The
original implementation tries to detect it by comparing values in hasChanged
but this fails if a new entity happens to have the same values as the old entity
its id was recycled from. In this case no diff is produced but the observer
does still serialize an AddEntity for the new entity which leads to
undefined component data on the deserializing end.

The same contamination can occur for any component array, not just the one
the removed entity had. Clearing only the shadow entry for the specific
component that was removed is therefore not sufficient — all shadow entries for
the removed EID must be cleared.

---

Expected Behaviour

After removeEntity(e) followed by addEntity() returning the same EID e,
the very next serialize([e]) call must emit data for every component array,
regardless of whether the current values match the stale values left by the
removed entity.

---

Proposed Fix

Expose a mechanism for the caller to inform the SoA serializer which EIDs were
freed so it can clear their shadow entries before the next diff pass.

This is implemented this by adding two APIs:

ObserverSerializerFunction.getRemovals(): Set<number>

Tracks entity IDs removed since the last call in onRemove(networkedTag) and
returns + clears that set. Entity-level tracking (keyed on the entity, not on
per-component removal events) is intentional — see point 3 above.

export type ObserverSerializerFunction = {
  (): ArrayBuffer;
  getRemovals: () => Set<number>;
};

SoASerializerOptions.getRemovals?: () => Set<number>

Before each diff pass, the serializer calls getRemovals() and clears all
shadow entries for every returned EID:

export type SoASerializerOptions = {
  diff?: boolean;
  getRemovals?: () => Set<number>;  // NEW
  // ...
};

const clearEntityShadow = (shadowMap: Map<any, any>, eid: number) => {
  for (const shadow of shadowMap.values()) {
    shadow[eid] = undefined;
  }
};

const serialize = (indices) => {
  if (diff && shadowMap && getRemovals) {
    for (const eid of getRemovals()) {
      clearEntityShadow(shadowMap, eid);
    }
  }
  // ... normal diff logic
};

SoASerializerFunction.clearEntity?(eid: number): void

A lower-level escape hatch for callers that manage removals manually, without an
ObserverSerializer.

Wiring in the caller

const observerSerialize = createObserverSerializer(world, Networked, components);
const serverSerialize = createSoASerializer(components, {
  diff: true,
  getRemovals: () => observerSerialize.getRemovals(),
});

---

Additional Footgun: Accidentally Serializing Non-Networked Entities

In my game I discovered a second failure mode that interacts with the same
underlying issue.

There's a server-only entity type (PendingUserCommand) that is not tagged
as Networked. It can end up being created with a recycled EID previously held
by Networked entity that was removed.

My mistake was passing getAllEntities(world) to the SoA serializer, which includes
non-Networked entities. The above fix makes this worse than usual.

1. getRemovals() correctly returns {X}, clearing shadow at X.
2. A non-Networked entity is created at EID X.
3. getAllEntities includes X — with stale component values (from an even older
   entity) which is serialized as a false diff.
4. The client has no idMap entry for X (deleted when original entitye was removed),
   so the deserializer falls back to using X as the raw array index.
5. In my case the Client array index X happens to be a live entity of a different
   type and it's coordinate value was overritten by stale data from entities back.

This is a caller-side bug — non-Networked entities should not be
included in the serialization call. But I think it might be worth emphasizing
in the documentation because discovering the cause was exteremely painful

---

Impact

Any application that:

• removes and re-creates entities (EID recycling occurs whenever entities are
  destroyed and new ones are allocated)
• uses { diff: true } SoA serialization
• has cases where a new entity's field value matches the old entity's stale value

…will silently drop component data for the recycled entity on the next
serialization pass, leaving the remote world with undefined component state.

The bug is probabilistic: it only manifests when the new value coincidentally
equals the stale value. This makes it hard to reproduce reliably in small tests
but guarantees it will appear in production given sufficient entity churn.