Add backtrack parser.
#220
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Feature
backtrack nis a parser that rewinds the input bynbytes, or fails if not enough client-uncommitted bytes are available. It is sorted with the expert, undocumented parser since it requires a good understanding of the input to be used safely.I do understand that this feature may seem dangerous, yet I think it is legitimate and useful in some contexts (see rationale below). I would definitely understand if it's deemed too tricky to integrate upstream, I'm fine using our pinned version.
Rationale
We use
Angstrom.scanto consume UTF-8 input. Thus, we must sometime read multiple bytes to get an actual unicode character. If we then try to reject that character by returningNone, only the last byte is pushed back to the input, losing some information and probably making the input malformed UTF-8.backtrackenables to roll back a few additional bytes to the start of the actual UTF-8 character. It is safe to use in such a case since we know there are at least that many uncommitted bytes to rollback.