Fixing overly verbose de/serialization on wire protocol

[chris-belcher]

Currently the project uses rust's serde to handle protocol messages. This is nice but sometimes creates larger-than-necessary messages. We should try to reduce the size of these messages as much as possible. Smaller messages really help with censorship-resistance, decentralization and privacy. The smaller messages can be harder to detect and cost less especially in perhaps third world countries or burner SIM cards where internet bandwidth is more limited.

The messages are defined in the file src/messages.rs

For example see this serialization of ProofOfFunding, which has a huge amount of wasted space. The funding transactions have witnesses which are serialized as lists of comma-separated numbers, which is pretty wasteful. A better way might be to somehow serialize those transactions in network format (i.e. as a big hex string)

{"method":"proofoffunding","confirmed_funding_txes":[{"funding_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"808cba6ad9ca4339cf6b7a1ce0dfc2419f6b71506b6c6e40f15c47dfcb4b4b00:1","script_sig":"","sequence":0,"witness":[[48,69,2,33,0,172,242,95,170,93,8,11,86,213,158,3,244,62,84,37,62,99,170,14,91,230,139,26,84,181,221,11,234,22,11,244,252,2,32,1,107,114,138,182,116,77,184,124,209,4,91,74,83,45,105,213,188,88,113,239,176,42,243,254,198,187,158,83,174,154,235,1],[3,144,16,150,236,5,184,239,225,112,70,71,85,136,90,215,212,140,54,95,173,116,37,81,95,93,220,173,92,10,124,26,36]]},{"previous_output":"7a7e61fdf221761d74bbb222b5e102a45e63e9bbd90dee0184547f6b738dd0a3:1","script_sig":"","sequence":0,"witness":[[],[48,69,2,33,0,156,21,218,170,167,68,188,186,74,137,111,212,60,47,129,45,48,237,20,216,23,109,135,75,57,33,210,209,31,71,14,153,2,32,104,200,75,77,227,75,108,139,42,233,21,185,203,123,58,164,230,29,75,187,166,240,95,155,109,35,108,213,61,245,186,207,1],[48,69,2,33,0,146,159,250,174,125,183,73,233,155,247,57,90,35,177,4,77,195,176,85,148,100,130,212,51,29,231,190,179,212,123,103,48,2,32,96,124,88,78,233,42,230,194,153,39,13,17,37,9,231,24,246,95,20,221,129,230,240,13,20,183,3,43,105,65,162,41,1],[82,33,2,88,167,91,153,45,237,19,0,19,207,44,105,7,41,12,144,244,199,43,224,163,22,253,229,48,87,200,57,189,142,13,76,33,3,218,156,42,131,43,176,85,199,3,144,246,93,71,253,125,108,8,15,149,119,112,82,219,205,107,9,169,161,200,2,116,79,82,174]]},{"previous_output":"bc078620097c4c037dbf7547ebd560d660d3c18352b81428c34120f98b3a1ab0:0","script_sig":"","sequence":0,"witness":[[],[48,68,2,32,20,231,123,87,145,9,205,101,118,63,240,2,87,184,201,245,151,29,92,145,55,246,88,105,18,244,16,8,104,44,50,54,2,32,84,32,171,21,64,133,10,189,74,60,214,134,49,17,136,31,62,26,176,113,2,119,168,4,158,100,45,89,214,238,65,130,1],[48,69,2,33,0,204,177,223,112,5,73,3,182,0,195,35,169,194,141,236,57,230,123,34,106,197,194,185,9,10,149,29,123,174,233,159,140,2,32,120,44,110,27,168,17,81,68,225,163,12,141,163,176,42,184,112,223,87,223,62,5,121,248,14,157,112,45,5,81,174,166,1],[82,33,3,195,10,155,69,54,207,10,76,207,35,20,76,78,199,140,221,12,219,229,163,40,217,69,92,151,11,181,75,46,228,139,109,33,3,239,193,123,235,77,216,86,221,165,234,216,52,121,122,249,251,247,222,197,95,202,118,209,161,7,77,215,37,50,78,58,41,82,174]]}],"output":[{"value":190015,"script_pubkey":"00203b181aaddd59457e45a336893e727d120d320ff1eb94260312c5039885e704e6"},{"value":1014560,"script_pubkey":"0014eefbe91835f348e5b14cc5ac36d37437678e15ab"}]},"funding_tx_merkleproof":"00000020268af582b0bb4bdb9b7515232649fab5023c4ae8723e4c2afbcd0bfedbe46346f5611e0ad84014830fc2009965988ba9ca7038237b4c1da454b919bad3c3db831187bb60ffff7f20030000000400000003a417b469a0e5df8478de553ae6228dbeaf96d33103885663875768c0b18bd851f99e42f5b87dc1c1e66ac6424ecec46a56e628c877569a6ba53e78e4cb0bba665c74288f9a9e3dac010745cc720a6d45a292db93ca7014067bec93d76716b628010b","multisig_redeemscript":"522102fe8ecc46c0d8bec30030670b6b021114fc0e843cc61622b8a3e7469934b3bb4221035cca74550e338599db8c5a5fa57b9cba0ae2301ffeeca79c21313b8d20b4fd0d52ae","multisig_key_nonce":"1caa072ecb693512de17731ea0f2c8b25e8902003b32915d2d7907d04aef9965","contract_redeemscript":"827ca9146ad908ce5acc2e74995d1894813b6a611b91a7a3876321033aa9f24c4984847649fecfd72dd28ce78b5a5019e14bd9653a5660c2857d75c9012051672103f0ca6f6e2a15f04a93521915144904a3977db9447c36b06deaec973c39dbec0100012868b2757b88ac","hashlock_key_nonce":"d9408d8d0a94f232436b9fb8ed8031d009be7517c16d6d4d8fc5a1471bf1a540"},{"funding_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"57f5d32ee574d98f28f66664224d2dbf74aa90e4471b351a46d8fdb9049bf142:1","script_sig":"","sequence":0,"witness":[[48,69,2,33,0,191,66,131,130,205,45,119,77,118,218,167,104,42,135,212,36,115,214,150,24,208,66,207,91,35,33,153,124,216,190,183,225,2,32,38,146,235,192,249,206,197,16,8,150,200,127,24,1,6,128,32,186,163,148,239,80,126,165,83,111,253,118,212,16,207,210,1],[3,87,136,122,207,251,59,161,89,254,157,20,132,220,248,42,63,179,92,229,209,119,114,139,208,20,253,87,250,151,77,171,19]]},{"previous_output":"dfa9293d4acddbaf2453d7edc134267ae8dc9ff6c771bff42553c2fff25d8b79:0","script_sig":"","sequence":0,"witness":[[],[48,69,2,33,0,163,16,136,197,229,117,105,117,75,94,38,226,206,63,71,181,212,183,81,149,27,210,128,2,210,134,246,110,177,149,179,54,2,32,87,126,236,101,27,32,208,229,74,48,212,208,51,161,241,248,31,51,169,72,119,158,163,232,98,176,180,119,156,104,141,47,1],[48,68,2,32,41,184,77,55,243,4,131,18,209,33,31,101,42,123,185,239,252,68,61,226,2,182,176,212,18,134,120,13,131,64,147,26,2,32,52,253,82,160,172,136,200,175,43,45,191,29,254,186,22,224,171,207,22,214,232,107,204,17,227,66,91,3,141,146,98,40,1],[82,33,2,221,3,13,249,198,138,98,24,226,152,104,211,143,92,127,90,120,160,8,244,139,95,89,163,33,143,28,72,43,199,125,42,33,2,239,216,184,91,173,92,64,172,255,8,201,82,253,250,234,189,144,146,44,77,214,95,40,81,238,150,39,42,127,205,208,79,82,174]]}],"output":[{"value":56031,"script_pubkey":"0020202d7a02d9e73e44b6a7bd1181a4e4af1f9ba8c47b52c13bfc1c1551f6aae5b4"},{"value":1124499,"script_pubkey":"00142badebf2a107c2dc3790a1d8b4891a6456796f73"}]},"funding_tx_merkleproof":"00000020268af582b0bb4bdb9b7515232649fab5023c4ae8723e4c2afbcd0bfedbe46346f5611e0ad84014830fc2009965988ba9ca7038237b4c1da454b919bad3c3db831187bb60ffff7f2003000000040000000395874728ea7dae24527f7c751f6e0572b33c052d753c6fed1024fa47dbdefa1f391278807864fb2249b468332b0c27f7def0f7eae1f2a84c17a9881c7c0e52df7c30bf78315a90d5190c32d34a551d1b78cedc75c3d741e621248b4bc2db4f01010d","multisig_redeemscript":"522102df73a047903f2ab0f018c6124fac49f46c1d8715f9831ffc6825253924ca29e121038c6007545a0961ba8e585e39e56c23dcaf8500a596a99912992f87431438b6c752ae","multisig_key_nonce":"e7a7277920661d81d735f17a39a72369c375fc76ca221013fa5e2a7eb962e107","contract_redeemscript":"827ca9146ad908ce5acc2e74995d1894813b6a611b91a7a38763210275936721be65b5704c6438a32cc29dfd67a5868db676e06d4ecc86b617eb0db0012051672103cb49c11f978d22154c1892d4d96448c13dc412efdcf494ea5c6066782130f48400012868b2757b88ac","hashlock_key_nonce":"460a50d9a721f9ffd5b37649a59e3208e31b4dc30ffb351024cc4f807f166ef6"},{"funding_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"aebd85476582ae34e74962f997ba62000eb9216d69ebf5c7fbf31316cb09225d:1","script_sig":"","sequence":0,"witness":[[],[48,69,2,33,0,173,5,212,154,49,31,137,190,92,69,29,6,251,252,129,248,244,29,33,69,196,198,87,49,236,151,189,31,195,84,30,21,2,32,111,188,195,214,128,11,163,10,199,151,94,43,129,222,93,195,117,87,22,211,59,134,203,248,121,190,105,80,215,131,64,151,1],[48,68,2,32,4,225,38,188,168,247,155,211,7,79,30,185,59,184,146,126,150,210,110,117,138,221,151,186,60,198,1,5,136,14,185,209,2,32,19,83,115,219,102,143,215,227,65,138,147,116,57,69,84,33,74,69,92,97,71,160,41,208,124,95,65,18,4,137,207,152,1],[82,33,2,223,133,159,135,251,230,7,93,219,208,58,170,0,209,28,64,106,145,199,148,161,122,220,215,201,134,140,150,231,232,186,80,33,3,94,67,74,57,209,194,13,111,19,118,169,244,210,78,239,70,249,230,68,140,176,143,230,249,89,1,157,239,86,186,128,220,82,174]]},{"previous_output":"95a2d692e3be31087f37fbea99dd1f85c578752eca4175fe68e16c6ff8e961c0:1","script_sig":"","sequence":0,"witness":[[48,69,2,33,0,131,230,212,181,135,248,216,238,158,180,50,127,163,198,118,84,109,129,204,225,47,52,135,239,123,163,79,24,79,195,197,45,2,32,119,10,11,109,220,104,181,239,81,60,39,214,193,16,101,198,202,191,235,134,137,199,18,132,252,210,244,114,15,188,81,64,1],[3,204,151,21,0,49,78,90,230,18,96,123,189,182,110,93,53,47,164,215,212,51,231,49,188,66,41,187,15,94,244,80,58]]}],"output":[{"value":253954,"script_pubkey":"002003124fabf5648b3d783f99c59ce34d460a9578dba1d45f404b7233d10c4d11a0"},{"value":1062426,"script_pubkey":"0014102a8b15b6ed6552c27fea7a7b4de8eb584ebfb7"}]},"funding_tx_merkleproof":"00000020268af582b0bb4bdb9b7515232649fab5023c4ae8723e4c2afbcd0bfedbe46346f5611e0ad84014830fc2009965988ba9ca7038237b4c1da454b919bad3c3db831187bb60ffff7f2003000000040000000395874728ea7dae24527f7c751f6e0572b33c052d753c6fed1024fa47dbdefa1f391278807864fb2249b468332b0c27f7def0f7eae1f2a84c17a9881c7c0e52df7c30bf78315a90d5190c32d34a551d1b78cedc75c3d741e621248b4bc2db4f010115","multisig_redeemscript":"522102bcaf6932f690616537e5283f27940d2c4dfa7e754e966b7c4b9f1fc7fb2cb16b2103e586e27b36f48b0885994ea14528b0c42a1dc2b181e92b154883e4d6c8359c4652ae","multisig_key_nonce":"ea3b2a108e05610ba5ccca2a544b62a8a40edb0184c968211ae433f43110e36c","contract_redeemscript":"827ca9146ad908ce5acc2e74995d1894813b6a611b91a7a3876321023afa07b7029313d32cd928fd3aedd6477681d582009548190cae947936991ad90120516721024bc3f49155c489ede98896c189abe2237eabe5525687215083b46b12225684a100012868b2757b88ac","hashlock_key_nonce":"57509a3ce6e183a7da6204483f118c3b2d9680e3528a7b9ee6205e4405d1494c"}],"next_coinswap_info":[{"next_coinswap_multisig_pubkey":"0336b46ed75c1c4e5a68e5f137ac48dfae12ccaf87d14ee2dfab19de8fd1e0f49a","next_hashlock_pubkey":"0367531813abe902473bda22c4318443607217d33e23b09c4a4df021a58ca14f5a"},{"next_coinswap_multisig_pubkey":"03d9e3192493f8c173b734dcc2c7c31ef5dfb50c0eb42c9074cda0a3e2ca8bb44f","next_hashlock_pubkey":"0360dc9b76a301e192098768a5d799004cc71b96c9d3dfddc5030ead117e9ba275"},{"next_coinswap_multisig_pubkey":"030cbcc691edbb1febd4a8e7640bd7d7b9ca0e6f32407f0170e1e5d233b4691702","next_hashlock_pubkey":"02931899d36052273a92488d9ea0f8dce889a807eb525404645105e359b8c968c1"}],"next_locktime":35}

Another instance of waste is using hex strings, and to me it seems base64 or base84 would be better for size. For example these two messages:

{"method":"receiverscontractsig","sigs":["304402203bfae22451977e8abd36c2f626ff3412dde1d4bd30b3e4a16388eb1b3cc2029802202ed10c1f8be11c9db4bb390360bc8ad7a260bab9907db35aa0ffd41e26c5012a","304402206099943ee887eec2fe8e6b872ee1eb1e8704b5d81b68ebfcced82822c265dbdc02201299b67ceb79cd17b86fd278ac1b1b6306fbd004b002bce9eddc0a3ac7e75bc7","3044022010f4abcdc7c462cca2093da1746039c266d450f90794b8d5d8ad15debf2b414102200ff9b31b4890a3e15082dfc5333eff20642c8a9db01594947c3b3b9ca200a81e"]}

{"method":"privatekeyhandover","swapcoin_private_keys":[{"multisig_redeemscript":"522102957cf49797b648f2de8d6393a728e93505b7e8b360191326fb21312a86e648cd210336b46ed75c1c4e5a68e5f137ac48dfae12ccaf87d14ee2dfab19de8fd1e0f49a52ae","key":"16184d12d87c88a84721bc2da80844d048d8157a2f20b953aec23a984b426345"},{"multisig_redeemscript":"522102155d2677b34a07dcb57b7deee6bd06ce3786dbca1b0d56e1f4b2d4f0e8ba714f2103d9e3192493f8c173b734dcc2c7c31ef5dfb50c0eb42c9074cda0a3e2ca8bb44f52ae","key":"8c31af1f1f47e32b870f1b354b6359e990acadec01e010df78443aa0bf36c04f"},{"multisig_redeemscript":"522102130819e8ef82c2dac6f104a87c561a7ed32537cbd2635219bb3aaebfaaeb9b9f21030cbcc691edbb1febd4a8e7640bd7d7b9ca0e6f32407f0170e1e5d233b469170252ae","key":"f5f1644ba11ebde84aed198c7447a244cd662f4dbf0def68a2afb529233d23b1"}]}

Another example is here, which also has "hash value" serialized as another comma-separated list of numbers.

{"method":"signsenderscontracttx","txes_info":[{"multisig_key_nonce":"1caa072ecb693512de17731ea0f2c8b25e8902003b32915d2d7907d04aef9965","hashlock_key_nonce":"d9408d8d0a94f232436b9fb8ed8031d009be7517c16d6d4d8fc5a1471bf1a540","timelock_pubkey":"03f0ca6f6e2a15f04a93521915144904a3977db9447c36b06deaec973c39dbec01","senders_contract_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"66ba0bcbe4783ea56b9a5677c828e6566ac4ce4e42c66ae6c1c17db8f5429ef9:0","script_sig":"","sequence":0,"witness":[]}],"output":[{"value":189015,"script_pubkey":"00206cdf8789b8ba64eb3eeca5f34e3f14d45cca3cda9316097a6365beb1b701151f"}]},"multisig_redeemscript":"522102fe8ecc46c0d8bec30030670b6b021114fc0e843cc61622b8a3e7469934b3bb4221035cca74550e338599db8c5a5fa57b9cba0ae2301ffeeca79c21313b8d20b4fd0d52ae","funding_input_value":190015},{"multisig_key_nonce":"e7a7277920661d81d735f17a39a72369c375fc76ca221013fa5e2a7eb962e107","hashlock_key_nonce":"460a50d9a721f9ffd5b37649a59e3208e31b4dc30ffb351024cc4f807f166ef6","timelock_pubkey":"03cb49c11f978d22154c1892d4d96448c13dc412efdcf494ea5c6066782130f484","senders_contract_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"df520e7c1c88a9174ca8f2e1eaf7f0def7270c2b3368b44922fb647880781239:0","script_sig":"","sequence":0,"witness":[]}],"output":[{"value":55031,"script_pubkey":"0020a0d54633f126fe99b6269fb28d4adcc0825199979dae75f625536db91d4725ce"}]},"multisig_redeemscript":"522102df73a047903f2ab0f018c6124fac49f46c1d8715f9831ffc6825253924ca29e121038c6007545a0961ba8e585e39e56c23dcaf8500a596a99912992f87431438b6c752ae","funding_input_value":56031},{"multisig_key_nonce":"ea3b2a108e05610ba5ccca2a544b62a8a40edb0184c968211ae433f43110e36c","hashlock_key_nonce":"57509a3ce6e183a7da6204483f118c3b2d9680e3528a7b9ee6205e4405d1494c","timelock_pubkey":"024bc3f49155c489ede98896c189abe2237eabe5525687215083b46b12225684a1","senders_contract_tx":{"version":2,"lock_time":0,"input":[{"previous_output":"014fdbc24b8b2421e641d7c375dcce781b1d554ad3320c19d5905a3178bf307c:0","script_sig":"","sequence":0,"witness":[]}],"output":[{"value":252954,"script_pubkey":"0020c1be327ce4a160f310ad59959a0131f1049faf17c830895548e239370a4b8fd1"}]},"multisig_redeemscript":"522102bcaf6932f690616537e5283f27940d2c4dfa7e754e966b7c4b9f1fc7fb2cb16b2103e586e27b36f48b0885994ea14528b0c42a1dc2b181e92b154883e4d6c8359c4652ae","funding_input_value":253954}],"hashvalue":[106,217,8,206,90,204,46,116,153,93,24,148,129,59,106,97,27,145,167,163],"locktime":40}

There is documentation here https://serde.rs/field-attrs.html which could be useful. The tag #[serde(with = "module")] is one possible method to change the serialization of the messages.

One thing we could consider is moving away from json serialization entirely and making everything be binary? Though theres downsides to that too.

On the other hand, this whole issue might be a case of premature optimization(?)

[mojoX911]

On the other hand, this whole issue might be a case of premature optimization(?)

I think its probably better to handle it now rather than to leave for future. Its clear that some tidying up is needed and we might need to try different things before coming up with a good approach.

Possible way that came to my mind.

• Define our own serialization/deserialzsation logic and use serde to link them as you mentioned.

• Its better to make stuffs binary and compact(?), I have seen other projects using bitcoin consensus type encoding with little modification (mostly endian-ness). Should we have something like that? Can serde already handle that? then we might not need to write our own.

• Writing our own might have added benefit of having future flexibility so better optimization can be done. We have to cover basic data types and then can export it to higher level structures as traits. Will be some work. But might make sense to get it done.

Also it seems rust-bitcoin already have core compatible encoding here https://github.com/rust-bitcoin/rust-bitcoin/blob/master/src/consensus/encode.rs, maybe something that we can directly use? We just need to implement the traits for our structs.

[mojoX911]

To further the discussion in this regard, I have made a trial encoding with rust-bitcoin here https://github.com/mojoX911/teleport-transactions/blob/cffba4d20c37f4a78af13eb5fc49bb44286e42b9/src/messages.rs#L96-L143.

It works, and there is a test here https://github.com/mojoX911/teleport-transactions/blob/cffba4d20c37f4a78af13eb5fc49bb44286e42b9/src/messages.rs#L286-L321

This can be done for other network messages also. Which will effectively compact the message size, and also will let us handle
allowed method logic more elegantly than string comparison as discussed #23 (comment).

Before proceeding further in this approach I would like to have agreement on using consensus encoding trait of rust-bitcoin for this purpose.

To me it seems easier than hand crafting our own encoding method, as all the primitive types are already covered in rust-bitcoin.

[chris-belcher]

Using rust-bitcoin's already-existing function is a good idea. Your code looks good I think.

However see my comment in #23 (comment). Using bytes instead of strings doesnt make the allowed method logic more elegant, and also doesnt make use rust's compiler for error checking, only using enums does that. So the two goals are orthogonal.

[mojoX911]

@chris-belcher agreed. I misunderstood the goal in #23 . Updated the PR as per your comment.

And that I think allows us to remove method flag from messages all together.

I will start working on the encoding of remaining structures.

[GeneFerneau]

One thing we could consider is moving away from json serialization entirely and making everything be binary?

There is a serde_cbor crate for serializing data in resource constrained environments. That way we could move to a more efficient serialization format without abandoning the benefits of serde.

[chris-belcher]

serde_cbor seems like almost exactly what we want I think(?)