Skip to content

Incorrect checksum calc for odd-sized buffers on big-endian #1044

@ghost

Description

On big-endian processors, the CalculateSum() func in core/utils/checksum.h will return an incorrect value when given a buffer with an odd number of bytes. This is because the last byte of the buffer is simply being promoted from 8 to 16 bits:

  // Add remaining 8-bit to the one's complement sum
  if (odd) {
    sum64 += *reinterpret_cast<const uint8_t *>(buf16);
  }

This works on LE, but on BE, the last byte should be left-shifted by 8 bits. (Recall, per the RFC, that the last odd byte should be treated as if it's the high-order byte of a 16-bit value in which the low-order byte is zero.)

For example, consider a five-byte buffer containing:

aabb ccdd ee

On BE, the expected adds should be:

     aabb
   + ccdd
   + ee00
    ~~~~~
    26598 = 659a (folded to 16 bits)

but will actually be:

     aabb
   + ccdd
   + 00ee
    ~~~~~
    17886 = 7887 (folded)

On LE, the adds should be (and will be):

     bbaa
   + ddcc
   + 00ee
    ~~~~~
    19a64 = 9a65 (folded)

Note that the LE result when byte-swapped equals the correct result from the BE calc, as expected.

This should give the correct result on both BE and LE:

   sum64 += be16_t(uint16_t(*reinterpret_cast<const uint8_t *>(buf16)) << 8).raw_value();

I can provide a test program to simulate this fix on an LE processor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions