Skip to content

Commit 7f63287

Browse files
committed
Allow a non-ASCII character to start a symbol
`identifier` still required `[a-zA-Z_]` in the leading position, so the non-ASCII members `ident_char` gained could only appear after an ASCII first character: `:foo日本語` lexed, but `:日本語` did not. Ruby restricts that position only by excluding ASCII digits -- every other identifier character may lead one, including a non-ASCII digit, so `:123` is a valid symbol literal. `rbs_next_char` never reports a non-ASCII character as `[0-9]`, so subtracting the ASCII digits from `ident_char` expresses exactly that rule: ident_start = ident_char \ [0-9]; `:123` and `{foo:123}` keep lexing as they did, because the leading position still bars `[0-9]`. The class is given a name because it states a rule about identifiers rather than about symbols, and the rules that lex an identifier take it up next. The leading `@`s fold into the same rule, which collapses the `:`, `:@` and `:@@` rules into one and leaves `identifier` with no use, so the definition goes away.
1 parent 4d56115 commit 7f63287

2 files changed

Lines changed: 667 additions & 750 deletions

File tree

0 commit comments

Comments
 (0)