-
Notifications
You must be signed in to change notification settings - Fork 160
Description
On big-endian processors, the CalculateSum() func in core/utils/checksum.h will return an incorrect value when given a buffer with an odd number of bytes. This is because the last byte of the buffer is simply being promoted from 8 to 16 bits:
// Add remaining 8-bit to the one's complement sum
if (odd) {
sum64 += *reinterpret_cast<const uint8_t *>(buf16);
}
This works on LE, but on BE, the last byte should be left-shifted by 8 bits. (Recall, per the RFC, that the last odd byte should be treated as if it's the high-order byte of a 16-bit value in which the low-order byte is zero.)
For example, consider a five-byte buffer containing:
aabb ccdd ee
On BE, the expected adds should be:
aabb
+ ccdd
+ ee00
~~~~~
26598 = 659a (folded to 16 bits)
but will actually be:
aabb
+ ccdd
+ 00ee
~~~~~
17886 = 7887 (folded)
On LE, the adds should be (and will be):
bbaa
+ ddcc
+ 00ee
~~~~~
19a64 = 9a65 (folded)
Note that the LE result when byte-swapped equals the correct result from the BE calc, as expected.
This should give the correct result on both BE and LE:
sum64 += be16_t(uint16_t(*reinterpret_cast<const uint8_t *>(buf16)) << 8).raw_value();
I can provide a test program to simulate this fix on an LE processor.