Internationally standardise BEVE

[SiebrenW]

I think BEVE is an excellent codec, having used it mostly for personal projects, one of the reasons my employers haven't shown interest is that it has no real footing outside of being a "hobby project".

We have been using stuff like protobuf, since it's an industry standard, but its reference implementation is a disgrace of generated C++ OOP-heavy getter-setter code. In automotive there are even standards using stuff like EXI (binary XML), because it's standardised, even though it's super convoluted and hard to work with, and few people even heard of it. Its only merit being that it is standardised.

The same is the case for REPE, a solution to real shortcomings of existing RPC protocols. I think industry adoption can be improved if it is somehow standardised.

My only idea of how standardisation would work initially is with an RFC or the like.

[stephenberry]

Thanks for your interest in BEVE, and I'm eager to get BEVE standardized, but I also want to make sure that it is exceptionally well done. The company I work for uses BEVE, but there are naturally questions about why this new format. I have reached out to two IETF members on different occasions, and while it hasn't been rejected, I was requested to look more carefully at CBOR to ensure this format wouldn't just be duplicating that format without reason.

CBOR offers a similar solution with similar performance BEVE. However, it is a much more complicated specification and relies on extensions for important features like typed arrays. So, many CBOR libraries don't implement features that are critical for performance.

To submit BEVE as an RFC I wanted to make certain I did my due diligence in comparing various aspects of CBOR and BEVE. I implemented probably the fastest CBOR implementation in the world in Glaze. So, I think I'm getting to the place where I can confidently propose BEVE as an RFC that deserves to live alongside formats like CBOR.

BEVE and CBOR in their current forms have nearly identical size and performance. BEVE is just more strict with things like typed arrays, requiring little endian. And, it aims for simplicity in implementation over shaving a few bytes here and there, as using a compression algorithm is common when size is critical, and complexity in the spec often leads to implementations that are slower.

There are a few additions that I've been considering for the BEVE spec that would significantly differentiate it from CBOR and enable significantly higher performance. These would make the argument for an RFC much stronger than it just being a simpler format. However, I think formats like JSON are excellent because they are simple, and it would have been bad to argue JSON shouldn't be created because YAML supports all that JSON provides.

Foremost, I'm looking into allowing padding to be encoded for typed arrays. This would enable zero copy access for arrays of types like floats and integers. Padding would be relative to a sentinel that could be encoded anywhere in the file, or multiple padding sentinels could be added.

I'm trying to use BEVE in production as much as possible right now, so that when it finally becomes an RFC it won't have any bad design choices locked in.

If you have any recommendations for changes to the spec or are willing to give feedback on an RFC proposal, let me know!

Sincerely,

Stephen

[stephenberry]

I did add more struct based benchmarks (I was just benchmarking typed arrays before) between BEVE and CBOR, and BEVE is consistently 25% faster. So, that is another reason for the spec.

[SiebrenW]

Maybe you can add a benchmark overview of these other formats as you do with json. I wouldn't know what library to compare some of these to, but comparing the various formats' speed to one another can help with a more informed choice when choosing the format.

Example being I offered glaze, with json, as an alternative to protobuf for our communication. They were afraid the encoding would be slower, but I showed them that without converting the protobuf datatypes to our own, it was still slower than glaze's json encoding/decoding. With an overview they would have had fewer doubts from the get-go and allowed me to use company time instead of my own. Their other argument against it was the absence of a schema or IDL, but since most our code is C++ anyway, that wasn't an issue in practice. We have some internal pilot applications now using it and I'm happy with the result so far.

I'm still interested in an IDL (I'll get to it, hopefully, or you will probably come up with something better if you do) but for pure C++ applications it's fine.