Skip to content

Commit 798d5b3

Browse files
committed
Stop the lexer reading past the end of a byte_range
`rbs_next_char` ends the input when `byte_pos == end_pos`. `rbs_skip` advances by a whole character, so a multibyte character starting before `end_pos` and ending after it steps over the boundary and equality never holds again — the lexer then reads to the end of the string. The realistic way to land inside a character is to pass a character offset where a byte offset is expected, the mistake #2945 fixed in `parse_inline_*_annotation`. `"日本語"` is 5 characters but 11 bytes, so offset 5 falls inside `本`: Parser.parse_type('"日本語" | Integer', byte_range: 0...5) #=> Types::Union spanning the whole input, rather than an error `require_eof: true` does not catch it, because the lexer really is at EOF by then. It needs a character to straddle the boundary, so ASCII-only input never hits it. Compare with `>=` so stepping over the boundary still ends the input.
1 parent 654954a commit 798d5b3

2 files changed

Lines changed: 24 additions & 1 deletion

File tree

src/lexstate.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ unsigned int rbs_peek(rbs_lexer_t *lexer) {
120120
}
121121

122122
bool rbs_next_char(rbs_lexer_t *lexer, unsigned int *codepoint, size_t *byte_len) {
123-
if (RBS_UNLIKELY(lexer->current.byte_pos == lexer->end_pos)) {
123+
if (RBS_UNLIKELY(lexer->current.byte_pos >= lexer->end_pos)) {
124124
return false;
125125
}
126126

test/rbs/type_parsing_test.rb

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1074,4 +1074,27 @@ def test_parse__byte_range_incorrect_in_euc_jp
10741074
assert_equal RBS::TypeName.parse("Foo"),
10751075
Parser.parse_type(euc, byte_range: 2...).name
10761076
end
1077+
1078+
def test_parse__byte_range_ending_mid_character
1079+
source = '"日本語" | Integer'
1080+
1081+
# `"日本語"` is 5 characters but 11 bytes, so a caller that passes a
1082+
# character offset where a byte offset is expected — the mistake fixed in
1083+
# #2945 — lands inside `本`, which spans bytes 4...7. The lexer used to
1084+
# step over the boundary and read the whole string instead.
1085+
assert_raises RBS::ParsingError do
1086+
Parser.parse_type(source, byte_range: 0...'"日本語"'.size)
1087+
end
1088+
1089+
# The byte offset the caller meant parses just that literal.
1090+
Parser.parse_type(source, byte_range: 0...'"日本語"'.bytesize, require_eof: true).tap do |type|
1091+
assert_instance_of Types::Literal, type
1092+
assert_equal "日本語", type.literal
1093+
end
1094+
1095+
# `end_pos` past the end of the buffer stays in bounds.
1096+
Parser.parse_type("Integer", byte_range: 0...9999).tap do |type|
1097+
assert_instance_of Types::ClassInstance, type
1098+
end
1099+
end
10771100
end

0 commit comments

Comments
 (0)