Return extra bytes from iterate and/or load

[scandey]

Related to earlier issue: #83 (comment)

Would be handy to be able to parse mostly well-formed but unfinished streams of packets and report or return the extra bytes at the end. This does not need to handle broken packets in the middle of the stream, just partial packets at the current end (where the last packet may or may not even have a length value).

[ddasilva]

This sounds reasonable at the surface, but I'm unsure as to what people would use it for. Can you help me understand the use case for this?

[scandey]

I imagine that this would be a useful part of a socket listener system for the cases where the socket has part of a packet available to be read. In utils.iter_packet_bytes, missing bytes (based on primary header) are reported. But as you pointed out in the earlier comment thread, it would be cleaner if iter reported or returned extra bytes (since that also covers the case of a packet including some of the header but missing a length field). When the socket reader is ready to read again, it could then seek back in buffer (or accept returned bytes from iter) to get the earlier "extra" bytes along with the new bytes.

[ddasilva]

This sounds reasonable. How do you think this compares to having utils.iter_packet_bytes() work from a file-like object whose read() method blocks when no new input is available? Thought this pattern doesn't make sense for utils.split_by_apid().

One thing we could do for utils.iter_packet_bytes() is have it return a custom object which populates an .extra_bytes attribute when it's complete. Currently, it just yeilds different byte objects and there is no good way to stick an extra return value into it. Something like this:

file_obj = open("something.tlm")
iter_obj= utils.iter_packets_bytes(file_obj)

for byte_stream in iter_obj:
    do_something(byte_stream)

print("These were the bytes left:", iter_obj.extra_bytes)