[pipeline] Handling of Serialization of Messages

[streamliner18]

Whether to use JSON, cPickle, or RabbitMQ Message header to handle serialization of message packs.

The current implementation with JSON has too many dirty hacks. In order to process as many types as possible, the current router and engine attempts to encode the message against multiple methods and notes down which method it used. This is particularly cumbersome and comes with performance penalty of doing JSON -> UTF-8 -> Base64 encoding back to back.

The initial proposal calls for serialization with cPickle, but this defeats the long term goal of moving beyond a Pythonic system for the router and potentially engine too (Even though there are ways to read a Pickle through Golang and other languages, these implementations are not mature).

A short summary of pros and cons of potential solutions here:

___	Pros	Cons
cPickle	Fast, Compatible with all data types	Unportable beyond Python
JSON	Highly compatible with other languages	No support for bytes etc, need external serializers
Header	Native way of doing things, most compatible	Still need additional serialization measures to store the messages

[streamliner18]

[D] Message Header for Metadata

[streamliner18]

[D] cPickle vs JSON