dot() in src/common.ts:12-16 accepts any non-word char as a right boundary (and consumes it). For token grammars where punctuation is structural, this returns a valid-looking prefix of a larger malformed token.
Examples of accepted prefixes today
1:2:3:4:5:6:7:8:9 → yields IPv6 1:2:3:4:5:6:7:8
00:1A:2B:3C:4D:5E:6F → yields six-pair MAC prefix
<jwt>.extra → yields the 3-segment JWT prefix
550e8400-e29b-41d4-a716-446655440000-dead → yields the UUID prefix
123-45-6789-00 → yields the SSN prefix
::ffff:192.0.2.128 → compressed parser returns partial ::ffff:192 instead of full IPv4-embedded IPv6
Relation to dot()
This is coupled to the fact that dot() also consumes the delimiter after an entity, which can hide an adjacent entity from the scanner (e.g. English+14155552671 loses the phone because + is eaten). A previous attempt to switch everything to a zero-width boundary() at once broke compound parsers (Relative time, Range) because they implicitly relied on child parsers swallowing trailing whitespace/delimiters. Reverted.
Proposal (careful migration, per-parser)
- Add a zero-width boundary primitive alongside
dot() (do not change dot() globally)
- For each affected parser, add format-specific continuation guards: IPv4 rejects adjacent digit/dot; IPv6 rejects hex/colon/IPv4-tail; MAC rejects hex or its separator; UUID rejects hex/hyphen; JWT rejects Base64URL chars/period; SSN rejects digit/hyphen
- Migrate compound parsers (Time.Relative, Range) deliberately with explicit space handling (
requireSpace/precededBySpace-style helpers) rather than relying on child consumption
- Sync/async scanner parity tests for adjacent entities
Tests to add
All the prefix cases above must produce no match (or the full embedded form for ::ffff:192.0.2.128), plus English+14155552671 → both language and phone.
Notes
Left-boundary checks are worth considering too. IPv4-embedded IPv6 (::ffff:192.0.2.128) is a correctness gap independent of boundaries.
dot()insrc/common.ts:12-16accepts any non-word char as a right boundary (and consumes it). For token grammars where punctuation is structural, this returns a valid-looking prefix of a larger malformed token.Examples of accepted prefixes today
1:2:3:4:5:6:7:8:9→ yields IPv61:2:3:4:5:6:7:800:1A:2B:3C:4D:5E:6F→ yields six-pair MAC prefix<jwt>.extra→ yields the 3-segment JWT prefix550e8400-e29b-41d4-a716-446655440000-dead→ yields the UUID prefix123-45-6789-00→ yields the SSN prefix::ffff:192.0.2.128→ compressed parser returns partial::ffff:192instead of full IPv4-embedded IPv6Relation to
dot()This is coupled to the fact that
dot()also consumes the delimiter after an entity, which can hide an adjacent entity from the scanner (e.g.English+14155552671loses the phone because+is eaten). A previous attempt to switch everything to a zero-widthboundary()at once broke compound parsers (Relative time, Range) because they implicitly relied on child parsers swallowing trailing whitespace/delimiters. Reverted.Proposal (careful migration, per-parser)
dot()(do not changedot()globally)requireSpace/precededBySpace-style helpers) rather than relying on child consumptionTests to add
All the prefix cases above must produce no match (or the full embedded form for
::ffff:192.0.2.128), plusEnglish+14155552671→ both language and phone.Notes
Left-boundary checks are worth considering too. IPv4-embedded IPv6 (
::ffff:192.0.2.128) is a correctness gap independent of boundaries.