Skip to content

Commit ef8e52a

Browse files
committed
Allow a non-ASCII character to start a name that is not a type name
The rules above this one already take a non-ASCII character anywhere but the first position, and the ones for symbols, instance variables and class variables take it there too. What is left is a bare identifier, and there the first character is not just a character: RBS reads its case to tell a class name from an interface name from an alias name. `rbs_next_char` reports every character outside ASCII as one and the same code point, so a rule cannot read that case on its own. The encoding could be asked, and its answer would be right -- the tables in `rbs_encoding.c` are a copy of prism's, and prism is what decides this in Ruby, so the two agree on every character in every encoding rbs knows. `TypeName#kind` is the problem. It reads `[[:upper:]]`, which is Onigmo's table, not prism's, and the two are different functions: they disagree about 891 characters, among them the full-width capitals of every EUC-JP and Shift_JIS variant, which Ruby does start a constant on. Agreeing would mean carrying prism's table on the Ruby side as well -- 662 code point ranges, 75 single-byte tables and the Japanese byte ranges, kept in step with the C copy forever. So RBS takes less than Ruby gives. A name whose kind is read from its first character has to open with ASCII, and every other name may open with whatever Ruby accepts. Those names get a token of their own: the parser takes `tNONASCIIIDENT` for a method name, a parameter name and a keyword, and nowhere a type name is read, so `class Foo日本語` parses and `class 日本語` does not. An incorrect `byte_range` lands on such a name, so the token it is reported against is a real one now instead of `ErrorToken`.
1 parent d726e2f commit ef8e52a

6 files changed

Lines changed: 541 additions & 544 deletions

File tree

include/rbs/lexer.h

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -72,18 +72,19 @@ enum RBSTokenType {
7272
kRETURN, /* return */
7373
kMODULESELF, /* module-self */
7474

75-
tLIDENT, /* Identifiers starting with lower case */
76-
tUIDENT, /* Identifiers starting with upper case */
77-
tULIDENT, /* Identifiers starting with `_` followed by upper case */
78-
tULLIDENT, /* Identifiers starting with `_` followed by lower case */
79-
tGIDENT, /* Identifiers starting with `$` */
80-
tAIDENT, /* Identifiers starting with `@` */
81-
tA2IDENT, /* Identifiers starting with `@@` */
82-
tBANGIDENT, /* Identifiers ending with `!` */
83-
tEQIDENT, /* Identifiers ending with `=` */
84-
tQIDENT, /* Quoted identifier */
85-
pAREF_OPR, /* [] */
86-
tOPERATOR, /* Operator identifier */
75+
tLIDENT, /* Identifiers starting with lower case */
76+
tUIDENT, /* Identifiers starting with upper case */
77+
tULIDENT, /* Identifiers starting with `_` followed by upper case */
78+
tULLIDENT, /* Identifiers starting with `_` followed by lower case */
79+
tNONASCIIIDENT, /* Identifiers starting with a character outside ASCII */
80+
tGIDENT, /* Identifiers starting with `$` */
81+
tAIDENT, /* Identifiers starting with `@` */
82+
tA2IDENT, /* Identifiers starting with `@@` */
83+
tBANGIDENT, /* Identifiers ending with `!` */
84+
tEQIDENT, /* Identifiers ending with `=` */
85+
tQIDENT, /* Quoted identifier */
86+
pAREF_OPR, /* [] */
87+
tOPERATOR, /* Operator identifier */
8788

8889
tCOMMENT, /* Comment */
8990
tLINECOMMENT, /* Comment of all line */

0 commit comments

Comments
 (0)