Repository navigation
feat: add Lookahead and StartTokens inspection APIs - #473
Pastalikek65 wants to merge 2 commits into
Conversation
RecoverTo(types...) enables skipping malformed input regions and resuming parse at the next token that can begin a registered production. It works inside repetition loops (struct slices parsed with @@* or @@+): when an element fails, the lexer scans forward for a synchronization token while never crossing structural literals (such as closing delimiters), then restarts the loop there. - recovery.go: startSet/nullable analysis over the node graph (literal, reference, strct, disjunction, sequence, capture, group, union), build-time resolution, and maybeRecover forward scan with lastRecovery progress guard. - nodes.go: hooks in group.Parse for both error and empty-match paths. - context.go/parser.go: recoverNode+structural sets threaded into parseContext. Full test suite green; fuzzed 45s / ~300k execs with no hang, panic or explosion. Closes alecthomas#342.
Expose the first-token-set analysis used by error recovery as public diagnostics: - Parser[G].Lookahead(example) maps each starting token to the disjunction alternative indices that can begin with it, for routing/autocompletion. - Parser[G].StartTokens(example) returns the deduplicated set of tokens that can begin a production. Both reuse the shared startSet/nullable analysis (recovery.go). Tokens are described as named references (<ident>) or quoted literals, with a :Type suffix when type-constrained. Lookahead reports an error for productions without a top-level disjunction. Closes alecthomas#107.
d6999a3 to
a29faee
Compare
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a29faee4a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| case *disjunction: | ||
| return n, true | ||
| case *strct: | ||
| return unwrapDisjunction(n.expr) |
There was a problem hiding this comment.
When example identifies an interface registered with Union, resolveProduction returns a *union, but this switch rejects it even though the API documentation explicitly lists registered unions as supported productions. For example, Lookahead((*Expr)(nil)) returns “does not have alternatives” instead of routing the union members; unwrap n.disjunction for this case.
Useful? React with 👍 / 👎.
| descs := startSetDescriptors(n, seen, p.caseInsensitiveTokens, p.typeNodes) | ||
| sort.Strings(descs) | ||
| return descs, nil |
There was a problem hiding this comment.
When multiple paths in a production have the same first token, startSetDescriptors emits that descriptor once per path and this code only sorts it. Thus a grammar such as @("a" | "a") returns ["\"a\"", "\"a\""], violating the documented deduplicated-set contract and forcing API consumers to deduplicate the result themselves.
Useful? React with 👍 / 👎.
| case *literal, *reference, *custom, *parseable, *negation, *lookaheadGroup: | ||
| return false |
There was a problem hiding this comment.
For a production beginning with a lookahead assertion, such as (?= "x") @Ident, the assertion consumes no token, but classifying lookaheadGroup as non-nullable prevents the sequence walker from considering the following consuming node. Consequently StartTokens returns an empty slice and RecoverTo registers no synchronization predicate for a production that can validly start with x.
Useful? React with 👍 / 👎.
| if p.structural[tok.Value] { | ||
| return false |
There was a problem hiding this comment.
When the token type is configured with CaseInsensitive, structural literals are still looked up by their exact grammar spelling here. For example, with an "END" block delimiter and input containing lowercase end, recovery from a malformed body scans across the delimiter to the next synchronization token, consumes that outer statement as part of the body, and ultimately reports a missing END error instead of recovering within the block.
Useful? React with 👍 / 👎.
Closes #107
Exposes the first-token-set analysis (originally built for the
RecoverTowork in #472) as two public inspection APIs onParser[G]:Token descriptors are human-readable: named token references render as
<ident>, literals as"if", and type-constrained literals as"if":Keyword.lookahead.go:Lookahead,StartTokens, sharedstartSetDescriptorsalongside the existingrecovery.goanalysis.Lookaheadroutes on toplevel disjunction alternatives, so a production with a single sequence (no|) reports an explicit error — that case is covered byStartTokens. Group starts (@("foo" | "bar")) are unioned into the start-token set.8 tests covering routing, group starts, no-disjunction, unknown type, nested productions, and dedup.
StartTokensoutput forscalars @Ident→["<ident>"].