#632 Buffer chunks to avoid partial matches in nearleyc#635
Open
naraen wants to merge 2 commits intokach:masterfrom
Open
#632 Buffer chunks to avoid partial matches in nearleyc#635naraen wants to merge 2 commits intokach:masterfrom
naraen wants to merge 2 commits intokach:masterfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue : #632
Cause : The StreamWriter receives tokens in chunks and passes it to the parser which in turn immediately passes it to a lexer. The default chunk size is 64KB. Token for a production rule could span chunks leading the lexer to match a partial token based on on the incomplete chunk.
Fix : Production rules in the nearley grammar are new line
\ndelimited. Passing only passing characters upto the newline should always result in a full match. StreamWriter buffers any characters in the chunk past the last newline prepends it to the next chunk.Implements the
Writable._finishto flush the buffer. This addresses grammar files that may not have a newline at the end of the file.