State synchronization across server nodes.

[bwasia]

Does Colyseus synchronize state across servers? Suppose if there's two servers Server A and Server B running on ports 2567 and 2568 respectively. They both share the same server codebase.

Please refer to the code snippets below. index.ts is the server code, followed by console outputs from Server A and B.

Server A and B connects to a Redis docker container running on port 6379. As shown in the console outputs, the state is not shared.

index.ts:

import "dotenv/config.js";
const port = process.env.COLYSEUS_PORT
    ? Number(process.env.COLYSEUS_PORT)
    : 2567;
import { RedisPresence, Room, Server } from "colyseus";
import { MapSchema, Schema, type } from "@colyseus/schema";

console.log("COLYSEUS_PORT", process.env.COLYSEUS_PORT);
console.log("REDIS_HOST", process.env.REDIS_HOST);

const serverOptions = {
    presence: new RedisPresence({
        host: process.env.REDIS_HOST,
        port: process.env.REDIS_PORT ? Number(process.env.REDIS_PORT) : 6379,
    }),
};

class Player extends Schema {
    @type("string") sessionId: string = "";
    @type("number") x: number = 0;
    @type("number") y: number = 0;
}

class RoomState extends Schema {
    @type({ map: Player }) players = new MapSchema<Player>();
}

class TestRoom extends Room<RoomState> {
    maxClients = 4;
    onCreate(options: any) {
        console.log("TestRoom created!", options);

        this.setState(new RoomState());

        this.onMessage("move", this.onMove.bind(this));
    }
    onJoin(client: any, options: any) {
        const player = new Player();
        player.sessionId = client.sessionId;
        this.state.players.set(client.sessionId, player);
    }
    onMove(client: any, coors: {x: number, y: number}) {
        console.log("TestRoom received coords from", client.sessionId, ":", coors);
        const player = this.state.players.get(client.sessionId);
        if (player) {
            player.x = coors.x;
            player.y = coors.y;
        }
        // print out all players coords
        this.state.players.forEach((player) => {
            console.log(player.sessionId, player.x, player.y);
        });
    }
}

const gameServer = new Server(serverOptions);
gameServer.define("testroom", TestRoom);
gameServer.listen(port);

Server A Console Output

$ NODE_ENV=development COLYSEUS_PORT=2567 ts-node src/index.ts
COLYSEUS_PORT 2567
REDIS_HOST 127.0.0.1

       ___      _
      / __\___ | |_   _ ___  ___ _   _ ___
     / /  / _ \| | | | / __|/ _ \ | | / __|
    / /__| (_) | | |_| \__ \  __/ |_| \__ \
    \____/\___/|_|\__, |___/\___|\__,_|___/
                  |___/

Multiplayer Framework for Node.js · Open-source

💖 Sponsor Colyseus on GitHub → https://github.com/sponsors/endel
🌟 Give it a star on GitHub → https://github.com/colyseus/colyseus
☁️  Deploy and scale your project on Colyseus Cloud → https://cloud.colyseus.io

Error: seat reservation expired.
    at WebSocketServer.onConnection (C:\..\state-sync-test\server\node_modules\@colyseus\ws-transport\build\WebSocketTransport.js:102:15)
    at WebSocketServer.emit (node:events:513:28)
    at WebSocketServer.emit (node:domain:489:12)
    at WebSocketServer.completeUpgrade (C:\..\state-sync-test\server\node_modules\ws\lib\websocket-server.js:358:5)
    at WebSocketServer.handleUpgrade (C:\..\state-sync-test\server\node_modules\ws\lib\websocket-server.js:271:10)
    at Server.upgrade (C:\..\state-sync-test\server\node_modules\ws\lib\websocket-server.js:107:16)
    at Server.emit (node:events:513:28)
    at Server.emit (node:domain:489:12)
    at onParserExecuteCommon (node:_http_server:903:14)
    at onParserExecute (node:_http_server:797:3)
TestRoom created! {}
TestRoom received coords from KzVK_m1Ou : { x: 2, y: 3 }
KzVK_m1Ou 2 3
TestRoom received coords from KzVK_m1Ou : { x: 66, y: 66 }
KzVK_m1Ou 66 66

Server B Console Output

$ NODE_ENV=development COLYSEUS_PORT=2568 ts-node src/index.ts
COLYSEUS_PORT 2568
REDIS_HOST 127.0.0.1

       ___      _
      / __\___ | |_   _ ___  ___ _   _ ___
     / /  / _ \| | | | / __|/ _ \ | | / __|
    / /__| (_) | | |_| \__ \  __/ |_| \__ \
    \____/\___/|_|\__, |___/\___|\__,_|___/
                  |___/

Multiplayer Framework for Node.js · Open-source

💖 Sponsor Colyseus on GitHub → https://github.com/sponsors/endel
🌟 Give it a star on GitHub → https://github.com/colyseus/colyseus
☁️  Deploy and scale your project on Colyseus Cloud → https://cloud.colyseus.io

TestRoom created! {}
TestRoom received coords from Fmy5Mx9hT : { x: 4, y: 5 }
Fmy5Mx9hT 4 5
TestRoom received coords from Fmy5Mx9hT : { x: 6, y: 7 }
Fmy5Mx9hT 6 7
TestRoom received coords from Fmy5Mx9hT : { x: 77, y: 77 }
Fmy5Mx9hT 77 77

[endel]

Hi @bwasia, when using multiple processes you must also provide a "publicAddress" config option for the server: https://docs.colyseus.io/scalability/#recommendation-using-a-load-balancer

This is required so the client-side makes the request directly to the node that created the room and has the room in memory. Each room and each process is stateful in Colyseus.

Hope this helps!

[bwasia]

hi @endel

Thank you for your reply.

So, when you say "Each room and each process is stateful," you mean stateful for that node, right? Am I correct to assume this?

So, if a room mutates its state, will the nodes in the cluster be informed about the mutation only if the publicAddress option is set?

[eugene-egorov]

There is no any sharing state across processes. To sync some state across the rooms you can use redis

[bwasia]

Thank you!

[joshestein]

Hey @endel, thank you for a superb library!

I am wondering if there are any examples of syncing state across servers with Redis? I have read the documentation but am running into some difficulties getting things correctly setup.

[zack1991]

Hi @endel ,

Thanks for your support my issue get resolved, the issue was related with package-lock.json. I deleted the package-lock.json and node_modules folder and then run nmp install after that i restarted my server and my issue get resolved.

[zack1991]

Hi @endel ,
My server is running perfectly when using client.js to connect to server. But when i am trying to join room with unity version 0.15.3 i am getting exception, But when running colyseus with single instance then it is working perfect. If you don't mind may you please help what is the problem, i will be thank full for that.

Error Log :-
Unable to connect to the remote server at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <9577ac7a62ef43179789031239ba8798>:0
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <9577ac7a62ef43179789031239ba8798>:0
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <9577ac7a62ef43179789031239ba8798>:0
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <9577ac7a62ef43179789031239ba8798>:0
at System.Runtime.CompilerServices.TaskAwaiter1[TResult].GetResult () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0 at Colyseus.ColyseusClient+d__251[T].MoveNext () [0x00200] in /Users/phonato-user/All Data/Bit Bucket/CamelCashCasino/Assets/Colyseus/Runtime/Colyseus/Models/ColyseusClient.cs:369

[endel]

Hi @zack1991 @VkDevPh, I've spent a few hours and couldn't reproduce this issue. (colyseus/colyseus-unity-sdk#215).

@VkDevPh also reported a "error CS0173: Type of conditional expression cannot be determined because there is no implicit conversion between 'method group' and '<null>'" (screenshot), which I also failed to reproduce.

Can you provide a reproduction scenario for these issues? (Unfortunately, I feel like wasting time when trying to reproduce issues like this without the full context.)

[hamaddo]

Hi @endel,
Do I need to specify publicAddress if I have two docker containers with colyseus app isntance (3000, 3001 ports), I'm using nest js, so here is how I init server:

    const presence = new RedisPresence({
      host: redisUrlParts.host,
      port: redisUrlParts.port,
      password: redisPassword,
    });

    const driver = new RedisDriver({
      host: redisUrlParts.host,
      port: redisUrlParts.port,
      password: redisPassword,
    });

    const server = new Server({
      transport: new WebSocketTransport({
        server: app.getHttpServer(),
      }),
      presence: presence,
      driver: driver,
      logger: new Logger('Colyseus'),
    });

I'm also using nginx as load balancer, here is my config:

upstream backend {
  ip_hash;
  server localhost:3000;
  server localhost:3001;
}

server {
    server_name your_domain.com;
    root /path/to/your/webroot;
    index index.html index.htm;

    location / {
        proxy_pass http://backend/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
        include proxy_params;
    }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /path/to/your/fullchain.pem; # managed by Certbot
    ssl_certificate_key /path/to/your/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    server_name your_domain.com;
    listen 80;
    listen [::]:80;

    location / {
        proxy_pass http://backend/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_read_timeout 86400s;
        proxy_send_timeout 86400s;
        include proxy_params;
    }
}

but I'm still getting error:

 ERROR [Colyseus] Error: seat reservation expired.
    at WebSocketServer.onConnection (/app/node_modules/@colyseus/ws-transport/build/WebSocketTransport.js:105:15)
    at WebSocketServer.emit (node:events:519:28)
    at WebSocketServer.completeUpgrade (/app/node_modules/@colyseus/ws-transport/node_modules/ws/lib/websocket-server.js:436:5)
    at WebSocketServer.handleUpgrade (/app/node_modules/@colyseus/ws-transport/node_modules/ws/lib/websocket-server.js:344:10)
    at Server.upgrade (/app/node_modules/@colyseus/ws-transport/node_modules/ws/lib/websocket-server.js:119:16)
    at Server.emit (node:events:519:28)
    at onParserExecuteCommon (node:_http_server:938:14)
    at onParserExecute (node:_http_server:839:3)

[endel]

Hi @hamaddo, the scalability docs have been improved on the upcoming version: https://deploy-preview-163--colyseus-docs.netlify.app/deployment/scalability

I would suggest enabling DEBUG=colyseus:* and checking if the traffic is coming as expected into your servers.
Colyseus Cloud have all these pieces sorted out, please consider subscribing if that works for you.