Commit ef8e52a
committed
Allow a non-ASCII character to start a name that is not a type name
The rules above this one already take a non-ASCII character anywhere but the
first position, and the ones for symbols, instance variables and class
variables take it there too. What is left is a bare identifier, and there the
first character is not just a character: RBS reads its case to tell a class
name from an interface name from an alias name.
`rbs_next_char` reports every character outside ASCII as one and the same code
point, so a rule cannot read that case on its own. The encoding could be asked,
and its answer would be right -- the tables in `rbs_encoding.c` are a copy of
prism's, and prism is what decides this in Ruby, so the two agree on every
character in every encoding rbs knows.
`TypeName#kind` is the problem. It reads `[[:upper:]]`, which is Onigmo's
table, not prism's, and the two are different functions: they disagree about
891 characters, among them the full-width capitals of every EUC-JP and
Shift_JIS variant, which Ruby does start a constant on. Agreeing would mean
carrying prism's table on the Ruby side as well -- 662 code point ranges, 75
single-byte tables and the Japanese byte ranges, kept in step with the C copy
forever.
So RBS takes less than Ruby gives. A name whose kind is read from its first
character has to open with ASCII, and every other name may open with whatever
Ruby accepts. Those names get a token of their own: the parser takes
`tNONASCIIIDENT` for a method name, a parameter name and a keyword, and
nowhere a type name is read, so `class Foo日本語` parses and `class 日本語`
does not.
An incorrect `byte_range` lands on such a name, so the token it is reported
against is a real one now instead of `ErrorToken`.1 parent d726e2f commit ef8e52a
6 files changed
Lines changed: 541 additions & 544 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
85 | | - | |
86 | | - | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
87 | 88 | | |
88 | 89 | | |
89 | 90 | | |
| |||
0 commit comments