Commit 7f63287
committed
Allow a non-ASCII character to start a symbol
`identifier` still required `[a-zA-Z_]` in the leading position, so the
non-ASCII members `ident_char` gained could only appear after an ASCII
first character: `:foo日本語` lexed, but `:日本語` did not.
Ruby restricts that position only by excluding ASCII digits -- every
other identifier character may lead one, including a non-ASCII digit, so
`:123` is a valid symbol literal. `rbs_next_char` never reports a
non-ASCII character as `[0-9]`, so subtracting the ASCII digits from
`ident_char` expresses exactly that rule:
ident_start = ident_char \ [0-9];
`:123` and `{foo:123}` keep lexing as they did, because the leading
position still bars `[0-9]`. The class is given a name because it states
a rule about identifiers rather than about symbols, and the rules that
lex an identifier take it up next.
The leading `@`s fold into the same rule, which collapses the `:`, `:@`
and `:@@` rules into one and leaves `identifier` with no use, so the
definition goes away.1 parent 4d56115 commit 7f63287
2 files changed
Lines changed: 667 additions & 750 deletions
0 commit comments