Any interest in a COBS Encoder/Decoder?

[fischermoseley]

Hello! I wanted to ask if there was any interest from the maintainers in adding a COBS encoder and decoder to this library.

I've been poking around with a very basic verison at fischermoseley/cobs that is absolutely in need of further development and definitely not ready for a PR - but if it would be an appropriate addition to amaranth-stdio I'm happy to develop it towards such. Otherwise I can keep it as a standalone project. No pressure either way, of course.

[whitequark]

Sounds good to me, COBS is very useful. We could prototype its use in Glasgow, perhaps.

[fischermoseley]

Great, thank you! I'll see if I can have a PR ready in the next few weeks.

[zyp]

I looked at your encoder and I have a couple of tips regarding improving throughput.

If I'm reading it right, your encoder will run at a fixed 50% throughput (not counting flow control), because it'll sit half the time in the SFZ state counting the number of bytes that doesn't need to be escaped, and half the time in the COB state outputting bytes.
The latency from counting bytes is unavoidable, but you can reach 100% throughput if you do the counting in parallel with the outputting.

For inspiration, here is a COBS encoder I wrote in Migen that I've been planning to eventually port to Amaranth.

In my implementation, a group is a set of contiguous unescaped bytes. The GroupSplitter block outputs the unescaped bytes to the data stream while counting and outputting the length to the len stream. Buffering the streams individually lets the GroupCombiner grab the len first to output the overhead byte, before outputting a group from the data stream.

The data buffer needs to be deep enough to hold a maximum sized group since the GroupCombiner won't start to consume a group before it has the length and the GroupSplitter won't output the length before all the data is out. Making the len buffer equally deep ensures that the GroupSplitter can keep busy while the GroupCombiner is still outputting even if you get a run of short groups following a long group. This ensures that the encoder is able to achieve and sustain full throughput regardless of the data it's fed.