Repository navigation
docs: add indentation-based parsing example - #471
Pastalikek65 wants to merge 1 commit into
Conversation
Parsing Python-like, block-structured languages requires turning leading whitespace into first-class INDENT/DEDENT tokens, which the built-in regex lexers cannot do. This adds a complete working example (custom Definition that emits INDENT/DEDENT, plus a nested "if" grammar using them) and lists it in the README. Closes alecthomas#20.
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e00d790a5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } | ||
| l.emit(tokEOL, "\n", lineNo+1, len(line)) | ||
| } | ||
| return l, nil |
There was a problem hiding this comment.
When the input ends while still inside an indented block, the loop exits without unwinding indents, so no closing DEDENT is emitted. A common valid input such as if x:\n print hello therefore fails with unexpected token "<EOF>" (expected <dedent>); emit one DEDENT for every remaining indentation level before returning the lexer.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
I prepared a two-file patch with regressions and raw observations addressing these three existing review findings at 5e00d790.
It flushes remaining DEDENTs at physical EOF, rejects unsupported characters at the current position, and tracks original spans with lexer.Position.Advance for one-based columns and byte offsets. The existing token vocabulary and blank-line/tab policy stay intact.
The identical 14 scenarios give 11 failures plus 3 passing controls before, and all 14 pass after. The documented go run ./indentation also keeps identical output. Tested with Go 1.26.9/macOS arm64 and the actual local-parent core selected by _examples/go.mod; patch replay matches the executed source tree.
This is Codex-authored support for kimjooyoon for the existing author to consider. The original findings remain credited to the earlier review. Only this example was tested; no whole-repository/race/hosted-CI or current-default-branch rebase result is claimed.
| m := tokenRe.FindStringIndex(rest) | ||
| if m == nil { | ||
| break | ||
| } | ||
| tok := rest[m[0]:m[1]] | ||
| rest = rest[m[1]:] |
There was a problem hiding this comment.
Because FindStringIndex searches anywhere in rest, any text before the next identifier or colon is discarded when the lexer advances to m[1], and an unmatched suffix is silently dropped by the m == nil branch. For example, print hello-world parses successfully as print hello world, changing the input's meaning instead of reporting an invalid -; explicitly consume allowed whitespace and return a lexer error when the next match does not start at offset zero.
Useful? React with 👍 / 👎.
| l.tokens = append(l.tokens, lexer.Token{ | ||
| Type: t, | ||
| Value: value, | ||
| Pos: lexer.Position{Filename: l.filename, Line: line, Column: column}, |
There was a problem hiding this comment.
lexer.Position.Column is one-based throughout this package, but every caller passes a zero-based byte index here, and Offset is never populated. Consequently diagnostics are shifted left—for example, the colon in print: is reported at column 5 rather than 6—and consumers capturing positions receive zero offsets; track the absolute offset and convert columns to the one-based position convention when emitting tokens.
Useful? React with 👍 / 👎.
Closes #20
Adds a complete, runnable example for parsing indentation-based (Python-like) grammars, since the built-in regex lexers can't express indentation as tokens.
_examples/indentation/main.go:lexer.Definitionthat measures leading whitespace per line and emits first-classINDENT/DEDENTtokens between linesifgrammar that consumes those tokens ("if" @Ident ":" EOL INDENT @@* DEDENT), including nested blocks_) are ignored by participle so token-consumers must be exported; and custom token types must not collide withlexer.EOF(-1) — a subtle bug whereDEDENTwas typed-1and silently treated as EOFREADME example table updated. Run with
go run ./indentationfrom_examples.