Skip to content

#632 Buffer chunks to avoid partial matches in nearleyc#635

Open
naraen wants to merge 2 commits intokach:masterfrom
naraen:master
Open

#632 Buffer chunks to avoid partial matches in nearleyc#635
naraen wants to merge 2 commits intokach:masterfrom
naraen:master

Conversation

@naraen
Copy link

@naraen naraen commented Apr 6, 2023

Issue : #632

Cause : The StreamWriter receives tokens in chunks and passes it to the parser which in turn immediately passes it to a lexer. The default chunk size is 64KB. Token for a production rule could span chunks leading the lexer to match a partial token based on on the incomplete chunk.

Fix : Production rules in the nearley grammar are new line \n delimited. Passing only passing characters upto the newline should always result in a full match. StreamWriter buffers any characters in the chunk past the last newline prepends it to the next chunk.

Implements the Writable._finish to flush the buffer. This addresses grammar files that may not have a newline at the end of the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant