BUG: parsing big json using input stream - bad hash/weakhash and name 🐛

[devechs]

the json is 64KB big - reading from the input stream

Json example :

  }, {
    "uid" : "SOME_UID",
    "status" : "UNKNOWN",
    "capabilities" : [ "

The "uid" was processed using generated parser - CompiledJson and next is "status"...
the code calls "fillNameWeakHash" which calls "calcWeakHash"..
it calculate the end of the name " this.nameEnd = this.currentIndex = ci + 1; "
and returns the value..
the issue happens in fillNameWeakHash - when looking for the colon ':' - calls getNextToken -> read()...
the read() calls -> this.prepareNextBlock() and the buffer loads the rest part of the buffer and then new values from the input stream.

after that in "bindSlow" calling "switch (reader.getLastHash()) {" whis uses the calculated name end value and applies it to the current buffer which has new values .. using the old value on refreshed buffer... the new buffer does not contain the json label at all... and the value of nameEnd is the old... so it calculate wrong values and the hash is not found at all, and getLastName calculates wrong value too...

if (reader.last() != ',') throw reader.newParseError("Expecting ',' for other mandatory properties");
else reader.getNextToken();
if (reader.fillNameWeakHash() != 735 || !reader.wasLastName(name_some_field)) {
	bindSlow(reader,....., 1);
	return;
}
.....

calcWeakHash

      if (this.stream != null) {
                while(ci < this.readLimit) {
                    b = this.buffer[ci];
                    if (b == 92) {
                        if (ci == this.readLimit - 1) {
                            return this.calcWeakHashAndCopyName(hash, ci);
                        }

                        ++ci;
                        b = this.buffer[ci];
                    } else if (b == 34) {
                        break;
                    }

                    ++ci;
                    hash += b;
                }

                if (ci >= this.readLimit) {
                    return this.calcWeakHashAndCopyName(hash, ci);
                }

                this.nameEnd = this.currentIndex = ci + 1;
            }
.....

 public final int fillNameWeakHash() throws IOException {
        int hash = this.calcWeakHash();
        if (this.read() == 58 || this.wasWhiteSpace() && this.getNextToken() == 58) {
            return hash;
....

public final int getLastHash() {
..
 i = this.tokenStart;
            for(int end = this.nameEnd - 1; i < end; hash *= 16777619L) {

this.currentIndex 4006 / this.readLimit 4058
reader 1 getLastName uid
reader 1 getLast 34
reader 1 getTokenStart 4006
reader 1 getCurrentIndex 4051
reader 1 buffer
.......ending of the buffer....

  }, {
    "uid" : "SOME_UID",
    "status" : "UNKNOWN",
    "capabilities" : [ "

... LOADING the NEW PART

reader 2 getLastName "96_AV_ " BUT EXPECTING "status"
reader 2 getLast 58
reader 2 getTokenStart 4051
reader 2 getCurrentIndex 1
reader 2 buffer
.......start of the buffer....
: "UNKNOWN",
"capabilities" : [

[zapov]

Tnx for the report.
Ideally there should be a test which shows the issue, as its unlikely it will be looked into otherwise.

[devechs]

`
private int prepareNextBlock() throws IOException {

    if (nameEnd == -1 || nameEnd < tokenStart) {

        int len = this.length - this.currentIndex;
        System.arraycopy(this.buffer, this.currentIndex, this.buffer, 0, len);
        int available = readFully(this.buffer, this.stream, len);
        this.currentPosition += (long) this.currentIndex;
        if (available == len) {
            this.readLimit = this.length - this.currentIndex;
            this.length = this.readLimit;
            this.currentIndex = 0;
        } else {
            this.readLimit = available < this.bufferLenWithExtraSpace ? available : this.bufferLenWithExtraSpace;
            this.length = available;
            this.currentIndex = 0;
        }

        return available;
    }

    // BUG FIX 

    final int lengthLabelTillEnd = this.nameEnd - this.tokenStart + 1;
    final int labelStart = this.tokenStart - 1;
    final int nextStart = this.currentIndex - labelStart;
    int len = this.length - labelStart;
    System.arraycopy(this.buffer, labelStart, this.buffer, 0, len);
    int available = readFully(this.buffer, this.stream, len);
    this.currentPosition += (long) this.currentIndex;
    if (available == len) {
        this.readLimit = this.length - this.currentIndex;
        if (readLimit < currentIndex) {
            this.length = len;
        } else {
            this.length = this.readLimit;
        }
    } else {
        this.readLimit = available < this.bufferLenWithExtraSpace ? available : this.bufferLenWithExtraSpace;
        this.length = available;
    }

    this.tokenStart = 1;
    this.nameEnd = lengthLabelTillEnd;
    this.currentIndex = nextStart;

    return available;
}`

cannot share the json because the data is private/protected

the fix is to copy the label when loading the new values in the buffer...
this is what works for me. @zapov