Support multiple serializers

[om26er]

Currently a single serializer needs to be provided to the WebSocketTransport class, it would be very useful to optionally allow to provide a list of serializers based on priority. If the router the "speaks" the first one it gets used, otherwise the next is tried.

It's more of an optimization but the binary serializers are much faster than json.

[konsultaner]

@om26er serializers are not negotiated in WAMP. Not sure if I will ever implement this. I will definitly accept a PR. If you need help with this, I could support you.

[om26er]

Right, we essentially need to send a list of serializers that our client talks in the initial handshake. The only user-facing change would be the addition of an API that takes serializers as a list instead of the current single serializer.

[konsultaner]

@om26er Could you get me more details on that? You want to extend the HELLO with a list of serializers and the CHALLENGE will tell the client what serializer to use?

[om26er]

@konsultaner the supported serializers have to be provided during the websocket handshake, they are negotiated before WAMP session is estabilished.

We need to change this code to take multiple serializers https://github.com/konsultaner/connectanum-dart/blob/master/lib/src/transport/websocket/websocket_transport_io.dart#L77

Once the connection is established the _socket.protocol can be queried to know which serializer was negotiated. Hence from that point onwards that serializer should be used for WAMP messaging.

The preference is from left to right, the router will return the first serializer that it supports.

[konsultaner]

@om26er _socket.protocol would have a value of wamp.2.json.msgpack.cbor, if all serializers are provided?

[konsultaner]

@oberstet does crossbar support that too? What do you think about it?

[om26er]

_socket.protocol would be wamp.2.json or wamp.2.msgpack or wamp.2.cbor depending on whichever sub-protocol (wamp.2.<serializer> in our case) is negotiated

This page might be of interest https://medium.com/@lancers/websocket-api-sec-websocket-protocol-subprotocol-header-support-277e34164537

Also afaik, Crossbar has that behavior and so does all autobahn libraries

[om26er]

And this line

_socket = await WebSocket.connect(_url, protocols: [_serializerType]);

would essentially look like

_socket = await WebSocket.connect(_url, protocols: [_serializerTypeCBOR, _serializerTypeMsgPack, _serializerTypeJson]);

[oberstet]

serializers are not negotiated in WAMP

yes, that is true. however, WAMP transports may have negotiation - as does WebSocket for example.

in code terms, both WAMP-WebSocket clients and servers maintain a list of serializers they support

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/websocket.py#L302

with WAMP-WebSocket clients, the client will send the list as WebSocket subprotocols: wamp.2.cbor, wamp.2.cbor.batched, ..

the WAMP-WebSocket server, that is the router, will select the first one from that list it supports

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/wamp/websocket.py#L198

RawSocket doesn't allow the client to provide a list for the server to choose from .. the client provides 1 serializer

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/twisted/rawsocket.py#L362

and the server must support that - otherwise it denies the client

https://github.com/crossbario/autobahn-python/blob/0090f300fa9e2d4a1ef2a44ee15241a9ad9ba4f3/autobahn/twisted/rawsocket.py#L289

If the router the "speaks" the first one it gets used, otherwise the next is tried.

the application can do that if it feels to. none of the autobahn libraries have that built in .. I think;) in any case, it's largely uneeded in my eyes, because:

• your router supports all serializers, doesn't it? ;)
• the client can be optimized and hard-coded for the best one

[oberstet]

writing above .. reminds me of "batched transports" ... which is a thing, and can bring quite some efficiencies. in particular on TLS. because there you want to shuffle multiple WAMP messages in a single TLS segment ... and the way TLS Python Twisted interacts .. anyways, details;)

[konsultaner]

@om26er @oberstet thanks for all that input. I was blind about the fact, that the _socket.protocol already is an array... I'll fix that.

[konsultaner]

RawSocket doesn't allow the client to provide a list for the server to choose from .. the client provides 1 serializer

@oberstet at the moment there is only msgpack and json support for raw sockets. 0x0001 for json and 0x0010 for msgpack if we added 0x0100 for cbor, the sending 0x0111 could mean that all 3 serializers are supported.

@om26er do you need raw socket at all?

[oberstet]

@konsultaner There is no negotiation in RawSocket by design. KISS;) The client must announce the one serializer it wants to talk, and the router accepts or denies. That's it. No need for ANDing of multiple values.

Having said that, I guess we might lack definitions for RAWSOCKET_SERIALIZER_IDs in the spec for RawSocket? Are we indeed?

AutobahnPython and Crossbar.io support RawSocket over TCP, TLS, UDS, Pipes and Serial on all supported Serializers ("RAWSOCKET_SERIALIZER_ID"):

• json: 1
• msgpack: 2
• cbor: 3
• ubjson: 4
• flatbuffers: 5

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/twisted/rawsocket.py#L399

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/twisted/rawsocket.py#L528

https://github.com/crossbario/autobahn-python/blob/91014f8bb4869ce123e631981d94b137abfaa68e/autobahn/wamp/serializer.py#L657

[konsultaner]

@oberstet @om26er ok, then well only have this feature for WebSockets. But to be honest, if crossbar supports all them why would some want to choose anyway?

the application can do that if it feels to. none of the autobahn libraries have that built in .. I think;) in any case, it's largely uneeded in my eyes, because:

• your router supports all serializers, doesn't it? ;)
• the client can be optimized and hard-coded for the best one

I don't really see a point. but I might do it if I get some time for that. @om26er If you are faster then me, feel free to send a PR.