Sequential decoding does not behave as expected

[BrianSipos]

Things to check first

• I have searched the existing issues and didn't find my bug already reported there

• I have checked that my bug is still present in the latest release

cbor2 version

5.8.0

Python version

3.12.3

What happened?

In cbor2 before version 5.8 I could construct a BytesIO buffer and sequentially decode items using a CBORDecoder object. Starting in 5.8 it appears that the decoder fp member is seeked to the end of the buffer and calling dec.fp.read() gives empty bytes because of that. In earlier versions, the decoder would leave the underlying fp cursor just past the last decoded CBOR item.

How can we reproduce the bug?

Here is a small script that succeeds with earlier versions and fails on line 9 with 5.8.0, where the cursor is reported as position 3 (EOF).

import cbor2
import io

data = bytes.fromhex('000102')

dec = cbor2.CBORDecoder(io.BytesIO(data))
assert 0 == dec.fp.tell()
assert 0 == dec.decode()
assert 1 == dec.fp.tell()
assert 1 == dec.decode()
assert 2 == dec.fp.tell()
assert bytes.fromhex('02') == dec.fp.read()

[andreer]

Hi @BrianSipos - sorry this is my fault, I added readahead to make decoding faster. You can set read_size to 1 restore the previous behavior:

import cbor2
import io

data = bytes.fromhex('000102')

dec = cbor2.CBORDecoder(io.BytesIO(data), read_size=1)
assert 0 == dec.fp.tell()
assert 0 == dec.decode()
assert 1 == dec.fp.tell()
assert 1 == dec.decode()
assert 2 == dec.fp.tell()
assert bytes.fromhex('02') == dec.fp.read()

cbor2/source/decoder.c

Lines 2416 to 2423 in fb4ee16

	":param read_size:\n"
	" the size of the read buffer (default 4096). The decoder reads from\n"
	" the stream in chunks of this size for performance. This means the\n"
	" stream position may advance beyond the bytes actually decoded. For\n"
	" large values (bytestrings, text strings), reads may be larger than\n"
	" ``read_size``. Code that needs to read from the stream after\n"
	" decoding should use :meth:`decode_from_bytes` instead, or set\n"
	" ``read_size=1`` to disable buffering (at a performance cost).\n"

[agronholm]

Yeah, this definitely needs to be fixed, as I imagine more than one user wants to decode a stream of CBOR objects.

[BrianSipos]

Since I do not really care about the fp directly, just about the decoder cursor, I will suggest two API changes to help this:

1. Provide a decoder tell method to determine the decoder view of the input cursor (independent of the fp)
2. Allow the read method to be given an argument -1 like other IO interfaces to mean "read the rest of the data" again from the decoder cursor not concerned with the fp state.

[oyvindronningstad]

I'm having the same issue.

Since the read_size parameter isn't in previous versions, I can't find a backwards compatible way to do the thing I want to. I want to check whether a given bytestring can be fully decoded as CBOR. I used to do it something like this:

f = BytesIO(obj)
can_be_decoded = False
try:
    cbor2.load(f)
    can_be_decoded = True
except (CBORDecodeValueError, CBORDecodeEOF):
    can_be_decoded = False
else:
    # Check if stream is exhausted
    if f.read(1) != b"":
        can_be_decoded = False

Is there a better way to do this, that works on both 5.7.1 and 5.8.0?

[oyvindronningstad]

Also, it seems read_size is not available in _decoder.py, though I might have missed something.

[oyvindronningstad]

Found this solution:

def strict_load(f):
    """Strict variety of cbor2.load that ensures all data is consumed."""
    dec = CBORDecoder(f)
    try:
        obj = dec.decode()
    except (CBORDecodeValueError, CBORDecodeEOF):
        pass
    else:
        try:
            dec.decode()
        except CBORDecodeEOF:
            return obj
        except CBORDecodeValueError:
            pass

    raise CBORDecodeValueError("Data not fully decoded.")

Is there anything better?