Pomelo data compression protocol

#Pomleo data compression protocol

In pomelo 0.3, we support two data compression method : route compression based on dictionary and message compression based on protobuf.

##Route compression based on dictionary ###The route field In Pomelo, we use the route field to determine the corresponding handler of a message. There are two kinds of route in Pomelo: The route used at server side, to dispatch a request to corresponding handler, and the route used at client side, the handle the messages published by server.

• The server side routes are generated by Pomelo at start, the content identify its corresponding handler. For example, the route "area.playerHandeler.attack" means it's handler is at server 'area', for the handler is 'playerHandler', and the method is 'attck'.

• The client side routes are used for to publishing messages, the client side used them the identify different messages, and the routes are defined by users.

When the data usage are not important, the overhead of route are acceptable. By in the bandwidth intensive environment, for example, on the mobile phones, the data traffic are much more expensive and limited. It will be necessary to compress the route field;

###The route compression in Pomelo In Pomelo, we use different strategy for the two kinds of route:

• For the route generated by the system, as 'area.playerHandler.attack', we give each route a unique id(16 bits short int).
• For the route defined by user, like 'onMove', 'onAttack'. because we don't know which routes are used, it's the duty of the developer to provide a dictionary for the route he/she used, and pomelo will assign a unique id the each routes.

When open the route compression function, when there is a message, the route will be replaced by it's id at sender side. At the server side, the id will be restore to the original route. And the procedure is invisible for the developers, you just open the dictionary function and provide a dictionary, all the job are done by pomelo. The below code is how to open the route compression function at app.js:

	app.set('connectorConfig',
		{
			connector : pomelo.connectors.hybridconnector,
			heartbeat : 3,
			useDict : true,
			handshake : function(msg, cb){
				cb(null, {});
			}
		});

The 'userDict : true' means to open the route compression. The default route compression will compress all the system generated routes, like 'area.playerHandler.move' etc.

To add the dictionary for user defined route, like 'onMove', 'onAttack', you need to add all the use defined routes at the /game-server/config/dictionary.json file, here is the example:

[
	"onDropItem",
	"onAttack",
	"onDied",
	"onMove",
	"onUpgrage",
	"onPickItem",
	"onRevive",
	"addEntities",
	"onRemoveEntities",
	"onPathCheckout"
]

At see above, the content of ditionary.json is an array of string, contends all the user defined routes need to be compressed.

##Data compress based on protobuf In pomleo 0.3, we use protobuf protocol the compress the message. Compare to other encode protocol like xml, json, the protobuf protocol has smaller size. In the project Lordofpomelo, the message size encode by protobuf can be 80% smaller than Json format on average.

###Protobuf protocol Protobuf are made by Goole, which is mainly used in rpc invoke and file encode. The Google protobuf has to parts : the protobuf protocol use binary encode, then the code generater based on protobuf meta data. The procedure of using google protobuf is as follow :

As in the above picture, to use the google protbuf, you have to generate code for each message need to be encode, and use the generated code to encode/decode the message. The disadvantage is obviously : it is complex to use, need to generate lots of code, change is very difficult.

So in pomelo protobuf, we use a differnt way to use the protobuf protocol.

###Pomelo-protobuf In Pomelo-protobuf we 原生的带有代码生成器的protobuf过于重量级，缺乏灵活性，任何消息的修改都会是一个非常重量级的操作，而这个在pomelo中应该是经常发生的。而随之带来的大量的代码生成会大大增加客户端的体积和部署难度。因此，我们没有采用生成代码的方式，而是根据proto文件的定义，对消息进行即时的解析。

####Pomelo protobuf实现 在pomelo中，我们实现了一个通用的protobuf编/解码器，以及一个proto文件解析器。通过分析proto文件内容，实现了对消息的编码/解码。这样，当修改/添加消息类型时，只需要修改对应的proto文件就可以了。具体的运行流程如下图：

从上图可以看出，与原生的protobuf生成代码的方式相比，pomelo中的解决方案要更将灵活，轻量。不需要生成任何代码，在运行时通过proto文件中对消息的定义，实现对消息的动态编码/解码功能。

####Proto 文件定义 原生的protobuf中，每一个消息与一个proto文件对应，而在生成编码/解码器之后，这个proto文件就不再被使用。 而在pomelo中，因为我们需要proto的内容来动态的对消息进行编码/解码，因此需要维护一个完整的protos信息表，一一对应的方式不但管理困难，也没有必要。因此将所有的proto定义放在一个json文件中，通过一个独立的key来进行区分，在pomelo中，key就是消息的route，我们的protos文件格式如下：

"onMove" : {
    "required uInt32 entityId" : 1,
    "message Path": {
      "required uInt32 x" : 1,
      "required uInt32 y" : 2
    },
    "repeated Path path" : 2,
    "required uInt32 speed" : 3
  },
  "onAttack" : {
    "required uInt32 attacker" : 1,
    "required uInt32 target" : 2
  }

这里的key就相当于原生protobuf中的proto文件中的消息名称，proto定义具体的语法也是按照protobuf标准来实现的，只是采用了对于js更容易解析的json形式。

在pomelo中，对于同样route的消息，如‘area.playerHander.attack'，在客户端和服务端的格式可能完全不同，这就意味着对于客户端的编码器和解码器对于同样route的消息需要不同的定义。因此，我们需要两套protos文件，server protos和client protos，具体的关系如下图：

####使用Protobuf 虽然protobuf的实现看上去十分复杂，但由于这一层对用户是完全透明的，使用会非常简单。用户只需要通过简单的两步定义就可以在原有的项目中开启protobuf功能。 首先，需要在connector组件上打开protobuf开关，在app.js中的配置如下：

	app.set('connectorConfig',
		{
			connector : pomelo.connectors.hybridconnector,
			heartbeat : 3,
			useProtobuf : true,
			handshake : function(msg, cb){
				cb(null, {});
			}
		});

实际上需要加入的就是“useProtobuf：true”这一项。当设置这一标识后，pomelo会在客户端握手时将protos内容同步到客户端，并默认开启protobuf压缩功能。 在protobuf功能开启用，用户还需要加入protos定义来实现对具体消息的编码/解码。protos文件默认在/game-server/config目录下，包括两个文件：serverProtos.json和clientProtos.json，分别表示服务端->客户端消息的protos和 服务端->客户端消息的protos。只要在其中加入有效的proto定义，就可以开启对应消息的protobuf编码功能。 ####与老项目的兼容性 Pomelo中的protobuf实现对原有项目是完全兼容的，你可以直接在老的项目中打开protobuf开关而不会引起任何问题。只是当proto定义是空的，默认所有的消息都不会经过protobuf压缩，而是采用默认的二进制编码进行传输。 当你相对某个消息进行protobuf编码时，只需要在对应的protos文件（serverProtos.json或clientProtos）中加入对应的protobuf项，pomelo在启动时就会自动识别并对消息进行压缩，而不会对其他未定义的消息产生任何影响。