Client -> Server replication

[Shatur]

Use cases

Originally, I wasn't sure about this feature.
I thought it's a very niche feature for VR games to avoid rollbacks (which might cause player nausea).
But I was wrong! For example, the Fusion networking library recommends using shared authority for mobile and WebGL games where security doesn't matter. This helps to avoid jittery gameplay on bad connections and reduces CPU load by avoiding rollbacks.

Prior art

Lightyear

Replication logic is completely independent from roles: any entity that represents a peer can be a ReplicationSender. This means that you can add replication with the same logic in these scenarios:

• Server to clients.
• Client to server.
• Client to other client(s) (for example, in a P2P scenario when a client joins mid-game and you want to replicate the current state of the game once, instead of the peer needing to replay all inputs since the beginning).

To select the replication direction, they have an authority system:

• The AuthorityBroker component can be added to a Server entity to signify that this entity is always aware of which peer has authority over an entity.
• Adding Replicate on an entity automatically grants the peer authority over it, unless another peer already had authority over it. (For example, C1 spawns an entity and replicates it to S. S receives the entity from replication, so it knows that C1 has authority over it. Then Replicate is added on S to replicate to other clients. Since C1 has authority, the authority is not given to S in this case.)
• In each link between two peers, only one of the peers has authority: that peer is allowed to send replication updates (and is currently not allowed to receive replication updates) on that link.
• Authority can be moved via messages:
  • The peer with authority can GiveAuthority.
  • Another peer can RequestAuthority.
  • The AuthorityBroker receives these messages, verifies whether authority is allowed to be transferred, and updates its bookkeeping of which peer has authority.

Bevy bundlication (before migration to Replicon)

Components marked for replication have a direction: client -> server or server -> client. Replication from the client happens only if the client has authority over the entity (similar to how we only replicate entities marked as Replicated). If the server disagrees, the replication is ignored.

Architecture options

Embrace Lightyear's approach

Make replication fully abstracted over roles using entities be defining the topology via entities. This allows having a shared implementation for server->client, client->server, and even client->client (P2P).

But I'm not sure if it's worth supporting P2P...
First, let's clarify the terminology, because I notice that people sometimes confuse P2P with deterministic replication. These things aren't related. Deterministic replication is when you exchange inputs. This can be used with both client-server and P2P. For example, Heroes of the Storm is client-server with deterministic replication. When you disconnect and reconnect, you wait for all inputs to replay. Client-server and P2P describe who connects to whom, basically a mesh network or a star network, respectively.

So why I'm not sure about supporting it?

• It's niche. I remember @Joy even said that client-server should be the only option in his networking proposal for Bevy. Also, I don't know any networking library or engine that supports both P2P and client-server. Even Photon doesn't support P2P; it's all client-server (notice there are no connections between clients).
• It makes things more complicated.
  • Because we support only client-server, we provide ClientState and ServerState, which is convenient. Based on these states, users can run special logic, such as displaying a 'connecting' message, despawning certain entities on network state changes, running preparation logic via OnEnter, using the built-in in_state run condition, etc.
  • With P2P, you don't have a server that is the single source of truth and need to implement distributed consensus models. Even for sending the initial state, you need to decide who will send it. Technically, you can treat one peer as a server, but then it's a client-server model with extra connections between clients.

If we decide not to support P2P, do we need to make our API fully entity-based to describe who sends data to whom? An API like this makes a lot of sense for Aeronet: you can have multiple active connections in a single app (analytics, gameplay, asset downloads, etc.). But with the client-server model, the client connects to a single server, and the server can have multiple clients.

Adjust the current API

We can try to extend the architecture to client->server replication:

• Move replication receive and send logic from the client and server modules into the shared::replication module (we already have a bunch of stuff there that is used on both client and server).
• ClientPlugin and ServerPlugin will just setup systems. We can make client->server replication optional via the client plugin toggle.
• Since replication can now go in two directions, we need additional channels for the client->server direction.
• We need to select which channels to use inside replication systems based on whether it's a client or server (this can be done based on the mentioned states).

This should require less changes.

Authority system

Independent of the architecture, we need some sort of permissions system for what to replicate and where. In the client-server model, the server is always the one that can give permissions.

I think it would be nice to have component-level granularity. For example, for something like a VR chat, it could be useful for clients to replicate Transform, but prevent changes to other components that might cause problems for other players.

Originally, I thought about introducing an Authority component with a bitmask that should be present on game entities and client entities. Users could associate components with bits, and an AND operation would result in the bits over which the client has authority. This is somewhat similar to how visibility works.
But then I realized that client entities aren't replicated. So on the client, you can't compute an AND from client and game entity masks. We could replicate the client entity to each client, but I think it's a bit ugly.

Another option is to introduce directions. This could work only because we have multi-component rules (otherwise it would be too limiting). For example, users would be able to call something like client_replicate_filtered::<Transform, With<Player>> or client_replicate_bundle::<(Transform, Name, PlayerInfo)>. We'll also need a special component that enables client->server replication, similar to Replicated: ClientAuthority { owner: Entity }. So if the entity contains ClientAuthority, we send all rules with the client->server direction.

I described ClientAuthority with a single client entity, so this way only a single client can have authority. I think in Lightyear you can also have only a single authority over an entity. However, I can imagine a use case where an entity owned by client A can be moved around by anyone, but only client A can change non-transform components 🤔
I'm not sure about this, though... Maybe, to keep things simple, we can go with a single client authority over an entity - mainly because I'm not sure how to implement this efficiently.

Lightyear also allows sending entities from clients. I think it could be useful for P2P, but it might be dangerous for client-server if you need security: clients can easily cause OOM on the server by spawning many entities. This also makes entity mappings a bit tricky. Right now, we map entities on the client on receive and before sending. But if clients can replicate new entities to the server, the server also needs to map entities. When the authority changes from client to server, the server should then send the entity without mappings.

If we disallow clients to spawn entities, no special handling will be needed because the client can replicate only entities that were previously sent by the server, which means they are present in its entity map and client can simply map them before sending. To give authority to a client, server will need to insert the mentioned ClientAuthority component (could be done via client request).

Also if we support only client-server, we won't need built-in events for authority management (since no need to send events to AuthorityBlocker). We can keep the API purely entity-based and server can insert components based on gameplay events.

---

All of this is just my initial thoughts on the subject, I would really appreciate your opinions 🙂

[cBournhonesque]

I agree with your thoughts around P2P.
Replicon was built with client-server in mind and it would be complicated to switch it now to support P2P; besides P2P is mostly deterministic replication.

Move replication receive and send logic from the client and server modules into the shared::replication module (we already have a bunch of stuff there that is used on both client and server).

Yes that would be great. It would be very convenient to move towards 'send' and 'receive' logic and the Client/Server plugins can just reuse subparts of it.

Lightyear also allows sending entities from clients. I think it could be useful for P2P, but it might be dangerous for client-server if you need security: clients can easily cause OOM on the server by spawning many entities. This also makes entity mappings a bit tricky. Right now, we map entities on the client on receive and before sending. But if clients can replicate new entities to the server, the server also needs to map entities. When the authority changes from client to server, the server should then send the entity without mappings.

That's an interesting idea. I guess the alternative would be the client sending some message to the server that says 'spawn an entity', and the server can do some validation on it? Maybe that would be equivalent to having an overridable spawn_fn in the ReplicationRules (similar to how the user can override DespawnFn, WriteFn, etc.) the user would be able to override the spawn logic to do some validation at that stage.

Not allowing clients to spawn entities would make handling entity-mapping easier; I do think it should be optional. I personally would want to directly spawn entities on a client.

I can already implement 95% of lightyear's authority handling in replicon right now, the only thing I would need is:

• a sender is not allowed to send replication updates on a link where they have no authority. (for example if Client 1 has authority, then if you consider the links between the server and clients 1 and 2, the server doesn't have authority on the link with client 1, and it has it on the link with client 2)
• a receiver is not allowd to apply replication updates on a link where they have authority.

For each link, the information of 'do we have authority or not' could be stored in ClientTicks.entities, which already contains a hashmap of replicated entities for all links.

Also if we support only client-server, we won't need built-in events for authority management (since no need to send events to AuthorityBlocker). We can keep the API purely entity-based and server can insert components based on gameplay events.

These events were inspired from the authority system in https://docs.coherence.io/manual/authority/authority-transfer

[Shatur]

Maybe that would be equivalent to having an overridable spawn_fn in the ReplicationRules

I thought about this! However, in order to validate the spawn request, users need access to the world. I could pass &mut World, but I don't think that's very ergonomic - and it might be a bit slow since data access isn't cached.
So instead, I imagined users would simply create an event (for example, SpawnFireball) instead of spawning via a command. Then, on the server, it can be validated inside an observer. It doesn't look like a lot of boilerplate, even if you don't want to validate the request 🤔

I personally would want to directly spawn entities on a client.

Maybe we can skip implementing it just for now to reduce the scope? This way we won't need to change mapping logic. Personally, I think that spawning via events is okay, but I'm open to discuss client-spawning. Maybe back to it later in a separate issue?

I can already implement 95% of lightyear's authority handling in replicon right now

Could you elaborate on this a bit more? I remember you were considering implementing it using a visibility system. Is that still the case? If so, do you see this as a temporary implementation until I get my hands on what I proposed above, or do you want the visibility-based implementation to become the primary one?

For each link, the information of 'do we have authority or not' could be stored in ClientTicks.entities, which already contains a hashmap of replicated entities for all links.

I thought about having a ClientAuthority component that holds the client entity that owns it. If the component is not present, then the server has authority by default.

However, I just realized that clients don't know their own client entity ID 🤔 Without this, even if we replicate ClientAuthority, clients won't know which entities they need to send updates for (unless client entities are replicated as well).

Maybe I need to think about this more. In Lightyear, this was a marker component, right? And But how did the server know which entity was controlled by whom?

These events were inspired from the authority system in https://docs.coherence.io/manual/authority/authority-transfer

My idea was similar to spawn validation: instead of having built-in events, users could handle this themselves by reacting to their own events. For example, the mentioned SpawnFireball event could spawn the fireball entity and immediately give authority to the spawner by inserting a component. However, it looks like my idea of component-based authority management wasn't very well thought out…

[mmannerm]

Just putting this thought here, even though it probably is wrong issue. Not sure if there is ambition to ever support server sharding, and replicating server to server on shard edges. More flexible replication targeting would be needed to support clients connecting to server, and server being able to be client to other servers, where authority can be swapped also when entities cross server boundaries.

[Shatur]

I'm not sure that servers that are close in a game world connect to each other. That would be a nightmare to maintain infrastructure-wise.
I'd expect there to be a single hub server to which shard servers are connected. That's how it works in WoW, at least.

[mmannerm]

Vanilla WoW is heavily layered and zone-based, where zone transitions are basically loading screens, and you can’t see or interact across zone borders. Other MMOs have varying degrees of “zoneless” designs, which requires ghosting of objects near the zone borders between the zone servers to provide illusion of no borders. Some games even have dynamic sharding where zones are not static and can adjust to player density in an area, while others use layering and other techniques. In grid-based zones, you may need to replicate object up to three other zones.

[Shatur]

This is correct. I'm saying zone servers unlikely connect to each other. I'd expect them to be connected to a single hub server 🤔
But I might be wrong since I never worked on an MMO.

[mmannerm]

Had to go and refresh my knowledge on current MMO architecture, as my experience was 25 years ago, when we built a worldserver, that was ghosting objects near boundaries with adjacent worldservers. I was thrown a bit off by your comment on hub server, as the term is “Replication Layer” what most seem to use for what you are talking about. And I can see the benefit of the replication layer. While people erroneously claim that it will help with O(N^2) problem, it does not as the nodes only need to connect to neighboring nodes. However, the fact that replication layer can act as a crash recovery and make authority transfer more consistent, is a big benefit. Most new MMOs like Star Citizen, New World, SpatialOS use the replication layer mechanism, while Ashes of Creation has reinvented technology that we did 25 years ago before WoW existed.

This though brings interesting though, how if any flexibility would help on the networking/replication layer to support these kind of patterns in future. But like I said in original comment, it may raise beyond the ambition level that this project should have.

[Shatur]

Thanks for the clarification!

Maybe it would be nice to make replication fully generic after all. I.e. allow any app to be configured as receiver and sender, like it's done in Lightyear 🤔
But this will make some things less convenient. When we assume it's either client or server, it makes the API more ergonomic. Especially around events. But I'll give it a thought. Thanks for the input.