Falling out of #3297, which added a fourth answer and then had to argue for it in a doc comment. Filing rather than widening that PR.
What is there
Three places in rust/ implement "skip a quoted literal and a /* ... */ comment", each written out by hand:
TypeScript has the same split: packages/parser/src/step-lexing.ts owns it over bytes, and #3297 added the decoded-string counterpart next to it in that same module, which is the shape this issue is asking for on the Rust side.
Why it matters, concretely
They do not agree on what an unterminated /* means, and that is not a detail. It decides how much of the file is thrown away:
|
unterminated /* |
geometry/src/processors/mod.rs |
consumes to end of input |
parser/src/step-lexing.ts (skipLexical) |
consumes to end of input, and stops the scan |
export/src/source_header.rs (#3297) |
not a comment at all, costs one / |
#3297 chose the third deliberately, because a header prescan that swallows every later record has lost the schema, while a tokenizer is already past the point of deciding one. That reasoning is written down in step-lexing.ts. It is a good reason for two answers. It is not a reason for three.
skip_lexical_at is pub(crate) in ifc_lite_export::source_header, which is why step_text.rs reaches across as crate::source_header::skip_lexical_at for a rule that is not about source headers.
Suggested shape
Move it to ifc-lite-core, which everything else already depends on, and have the other two call it. Naming the unterminated case as an explicit choice at the call site rather than a property of the copy you happened to reach for is most of the value.
One thing I checked and it is NOT a bug
While looking at this I thought in_string = !in_string in geometry/src/processors/mod.rs mishandled the '' escape. It does not. A doubled apostrophe toggles the flag twice and nets to a no-op, which is the same technique the older find_unquoted documented. Recording it so nobody else spends the time.
Not urgent
Nothing here is a live defect. It is the condition that produced #3284, one layer down: several copies of one rule, each documenting itself as matching the others.
Falling out of #3297, which added a fourth answer and then had to argue for it in a doc comment. Filing rather than widening that PR.
What is there
Three places in
rust/implement "skip a quoted literal and a/* ... */comment", each written out by hand:rust/core/src/parser/scanner.rsrust/geometry/src/processors/mod.rsrust/export/src/source_header.rs(skip_lexical_at, added by fix(parser,export): a comment in a STEP header cost the whole header, in both halves (#3284) #3297)TypeScript has the same split:
packages/parser/src/step-lexing.tsowns it over bytes, and #3297 added the decoded-string counterpart next to it in that same module, which is the shape this issue is asking for on the Rust side.Why it matters, concretely
They do not agree on what an unterminated
/*means, and that is not a detail. It decides how much of the file is thrown away:/*geometry/src/processors/mod.rsparser/src/step-lexing.ts(skipLexical)export/src/source_header.rs(#3297)/#3297 chose the third deliberately, because a header prescan that swallows every later record has lost the schema, while a tokenizer is already past the point of deciding one. That reasoning is written down in
step-lexing.ts. It is a good reason for two answers. It is not a reason for three.skip_lexical_atispub(crate)inifc_lite_export::source_header, which is whystep_text.rsreaches across ascrate::source_header::skip_lexical_atfor a rule that is not about source headers.Suggested shape
Move it to
ifc-lite-core, which everything else already depends on, and have the other two call it. Naming the unterminated case as an explicit choice at the call site rather than a property of the copy you happened to reach for is most of the value.One thing I checked and it is NOT a bug
While looking at this I thought
in_string = !in_stringingeometry/src/processors/mod.rsmishandled the''escape. It does not. A doubled apostrophe toggles the flag twice and nets to a no-op, which is the same technique the olderfind_unquoteddocumented. Recording it so nobody else spends the time.Not urgent
Nothing here is a live defect. It is the condition that produced #3284, one layer down: several copies of one rule, each documenting itself as matching the others.