Commit 4d56115
Define
Listing the characters an identifier may contain left no room for
non-ASCII ones. `rbs_next_char` folds every non-ASCII character into a
single code point, and naming that code point in the grammar would tie
the grammar to a value the C side itself calls dummy data.
Subtracting instead states Ruby's own rule
#define is_identchar(p,ptr,end,enc) \
(ISALNUM(*(ptr)) || (*(ptr)) == '_' || !ISASCII(*(ptr)))
without enumerating any punctuation:
ident_char = . \ ([\x00-\x7F] \ [a-zA-Z0-9_]) \ [\uFFFD];
The parentheses matter. `\` is left-associative, so without them this
would subtract the whole ASCII range and never add the word characters
back.
`\uFFFD` is the sentinel `rbs_next_char` reports for a byte that is
invalid under the active encoding. Keeping it out of the class is what
makes such a byte still surface as an ErrorToken instead of being
absorbed into a token, as it already does for `#` comments.
An identifier can now hold a non-ASCII character in any position but the
first: `MI_CONSTANTE_Ñ` is one tUIDENT rather than a tUIDENT followed by
an ErrorToken. A leading non-ASCII character still has no rule, because
nothing yet decides whether it starts a constant or a local.
src/lexer.c grows by 112 lines: two comparisons appended to each of 45
identifier-continuation states, and one more backtracking point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>ident_char by what it excludes1 parent 588ef6a commit 4d56115
2 files changed
Lines changed: 1161 additions & 1043 deletions
0 commit comments