Skip to content

Commit 4d56115

Browse files
soutaroclaude
andcommitted
Define ident_char by what it excludes
Listing the characters an identifier may contain left no room for non-ASCII ones. `rbs_next_char` folds every non-ASCII character into a single code point, and naming that code point in the grammar would tie the grammar to a value the C side itself calls dummy data. Subtracting instead states Ruby's own rule #define is_identchar(p,ptr,end,enc) \ (ISALNUM(*(ptr)) || (*(ptr)) == '_' || !ISASCII(*(ptr))) without enumerating any punctuation: ident_char = . \ ([\x00-\x7F] \ [a-zA-Z0-9_]) \ [\uFFFD]; The parentheses matter. `\` is left-associative, so without them this would subtract the whole ASCII range and never add the word characters back. `\uFFFD` is the sentinel `rbs_next_char` reports for a byte that is invalid under the active encoding. Keeping it out of the class is what makes such a byte still surface as an ErrorToken instead of being absorbed into a token, as it already does for `#` comments. An identifier can now hold a non-ASCII character in any position but the first: `MI_CONSTANTE_Ñ` is one tUIDENT rather than a tUIDENT followed by an ErrorToken. A leading non-ASCII character still has no rule, because nothing yet decides whether it starts a constant or a local. src/lexer.c grows by 112 lines: two comparisons appended to each of 45 identifier-continuation states, and one more backtracking point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 588ef6a commit 4d56115

2 files changed

Lines changed: 1161 additions & 1043 deletions

File tree

0 commit comments

Comments
 (0)