Skip to content

Commit 2cd3c5e

Browse files
johnsoncodehkjohnsoncodehk
committed
gen-tm: recover a JSX tag whose < and name are split across lines (fixes TSX #825)
`<div>\n <\n span className="foo">…` — a `<` alone on one line, the tag name `span` on the next. Valid JSX (tsc: nested JsxElements) but a TextMate `begin` is single-line, so the tag-open `(<)\s*name` can't span the break; both grammars scoped `span` as `meta.jsx.children` (plain text). Add a children-only multi-line tag-open `jsx-element-multiline`, included last in `#jsx-children`: `begin: (<)(?=\s*$)` opens only on a lone `<` reaching end-of-line. Inside JSX children a bare `<` is unambiguously a tag opener (tsc rejects a stray `<` in children), so this is safe — and it lives ONLY inside `#jsx-children`, never at expression-start, so a split comparison `a <\n b` or generic `f<\n T>()` outside JSX is never reached (verified: both stay non-tags). Inner patterns recover the name (namespace/member/ intrinsic/component), type-args and attributes on later lines; a child sub-region confines post-`>` content to children. Agnostic — built from derived primitives (nameRe, nameCaptures, selfCloseTok/closeTok), no hardcoded tokens. TSX ledger 10->11/11 (only-Monogram win; official still misses it). agnostic 8/8, tsx-highlight 56/56, tsx/jsx-conformance clean; TS/JS/HTML/Vue byte-identical.
1 parent 47d0326 commit 2cd3c5e

5 files changed

Lines changed: 312 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ The same question, every language at once: take the bugs reported against each *
4646

4747
<!-- issues:start -->
4848
<!-- generated by `npm run bench:issues` — do not edit by hand -->
49-
_Each hand-written **official** grammar vs Monogram's **derived** one, on the bugs filed against it: **TypeScript 26/27** (official 9/27) · **TSX 10/11** (official 6/11) · **HTML 20/20** (official 13/20) · **Vue 19/19** (official 15/19). Per-issue detail below — auto-generated by `npm run bench:issues`._
49+
_Each hand-written **official** grammar vs Monogram's **derived** one, on the bugs filed against it: **TypeScript 26/27** (official 9/27) · **TSX 11/11** (official 6/11) · **HTML 20/20** (official 13/20) · **Vue 19/19** (official 15/19). Per-issue detail below — auto-generated by `npm run bench:issues`._
5050

5151
#### TypeScript
5252
| issue | Monogram | official |
@@ -93,7 +93,7 @@ _Each hand-written **official** grammar vs Monogram's **derived** one, on the bu
9393
| [#979](https://github.com/microsoft/TypeScript-TmLanguage/issues/979) — `const` modifier on a type parameter in `.tsx` | ✓ | · |
9494
| [#1042](https://github.com/microsoft/TypeScript-TmLanguage/issues/1042)/[#990](https://github.com/microsoft/TypeScript-TmLanguage/issues/990) — default generic arrow function in `.tsx` | ✓ | · |
9595
| [#627](https://github.com/microsoft/TypeScript-TmLanguage/issues/627) — member-expression JSX tag name | ✓ | · |
96-
| [#825](https://github.com/microsoft/TypeScript-TmLanguage/issues/825) — `<` and tag name on separate lines | · | · |
96+
| [#825](https://github.com/microsoft/TypeScript-TmLanguage/issues/825) — `<` and tag name on separate lines | ✓ | · |
9797

9898
<details><summary>… and 6 more both grammars already handle (✓ / ✓)</summary>
9999

‎javascriptreact.tmLanguage.json‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,9 @@
258258
{
259259
"include": "#jsx-fragment"
260260
},
261+
{
262+
"include": "#jsx-element-multiline"
263+
},
261264
{
262265
"include": "#jsx-expression"
263266
},
@@ -510,6 +513,111 @@
510513
}
511514
]
512515
},
516+
"jsx-element-multiline": {
517+
"name": "meta.tag.js.jsx",
518+
"begin": "(<)(?=\\s*$)",
519+
"beginCaptures": {
520+
"1": {
521+
"name": "punctuation.definition.tag.begin.js.jsx"
522+
}
523+
},
524+
"end": "(?:(/>)|(</)\\s*(?:(?:((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(:))?(?:((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?:\\.(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)+)|([[:lower:]][-[:alnum:]]*)|([_$[:upper:]][-_$[:alnum:].]*|(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?:[-.][_$[:alnum:]]+)+))(?<!\\.|-))?\\s*(>))",
525+
"endCaptures": {
526+
"1": {
527+
"name": "punctuation.definition.tag.end.js.jsx"
528+
},
529+
"2": {
530+
"name": "punctuation.definition.tag.begin.js.jsx"
531+
},
532+
"3": {
533+
"name": "entity.name.tag.namespace.js.jsx"
534+
},
535+
"4": {
536+
"name": "punctuation.separator.namespace.js.jsx"
537+
},
538+
"5": {
539+
"patterns": [
540+
{
541+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?=\\s*\\.)",
542+
"name": "variable.other.object.js.jsx"
543+
},
544+
{
545+
"match": "\\.",
546+
"name": "punctuation.accessor.js.jsx"
547+
},
548+
{
549+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*",
550+
"name": "support.class.component.js.jsx"
551+
}
552+
]
553+
},
554+
"6": {
555+
"name": "entity.name.tag.js.jsx"
556+
},
557+
"7": {
558+
"name": "support.class.component.js.jsx"
559+
},
560+
"8": {
561+
"name": "punctuation.definition.tag.end.js.jsx"
562+
}
563+
},
564+
"patterns": [
565+
{
566+
"begin": "(?<=>)",
567+
"end": "(?=</)",
568+
"contentName": "meta.jsx.children.js.jsx",
569+
"patterns": [
570+
{
571+
"include": "#jsx-children"
572+
}
573+
]
574+
},
575+
{
576+
"begin": "\\s*(?:((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)(:))?(?:((?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?:\\.(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*)+)|([[:lower:]][-[:alnum:]]*)|([_$[:upper:]][-_$[:alnum:].]*|(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?:[-.][_$[:alnum:]]+)+))(?<!\\.|-)",
577+
"beginCaptures": {
578+
"1": {
579+
"name": "entity.name.tag.namespace.js.jsx"
580+
},
581+
"2": {
582+
"name": "punctuation.separator.namespace.js.jsx"
583+
},
584+
"3": {
585+
"patterns": [
586+
{
587+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*(?=\\s*\\.)",
588+
"name": "variable.other.object.js.jsx"
589+
},
590+
{
591+
"match": "\\.",
592+
"name": "punctuation.accessor.js.jsx"
593+
},
594+
{
595+
"match": "(?:[a-zA-Z_$\\p{L}\\p{Nl}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})(?:[a-zA-Z0-9_$\\p{L}\\p{Nl}\\p{Nd}\\p{Mn}\\p{Mc}\\p{Pc}]|\\\\u[0-9a-fA-F]{4}|\\\\u\\{[0-9a-fA-F]+\\})*",
596+
"name": "support.class.component.js.jsx"
597+
}
598+
]
599+
},
600+
"4": {
601+
"name": "entity.name.tag.js.jsx"
602+
},
603+
"5": {
604+
"name": "support.class.component.js.jsx"
605+
}
606+
},
607+
"end": "(>)|(?=/>)",
608+
"endCaptures": {
609+
"1": {
610+
"name": "punctuation.definition.tag.end.js.jsx"
611+
}
612+
},
613+
"patterns": [
614+
{
615+
"include": "#jsx-attributes"
616+
}
617+
]
618+
}
619+
]
620+
},
513621
"jsx-fragment": {
514622
"name": "meta.tag.js.jsx",
515623
"begin": "(<)\\s*(>)",

‎src/gen-tm.ts‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1146,6 +1146,14 @@ function generateJsxPatterns(langName: string, identRegex: string, jsx: JsxInfo,
11461146
{ include: '#jsx-self-closing-element' },
11471147
{ include: '#jsx-element' },
11481148
{ include: '#jsx-fragment' },
1149+
// A tag whose `<` is alone at end-of-line, its NAME on a following line
1150+
// (#825). In children a lone `<` is unambiguously a tag opener (a bare `<`
1151+
// in JSX children is otherwise invalid — tsc rejects `<div>a < b</div>`),
1152+
// so this multi-line opener can recover the split tag the single-line
1153+
// `(<)\s*name` begins miss. Listed AFTER the single-line elements so a
1154+
// normal `<span …>` on one line keeps its existing tokenization; it only
1155+
// fires when the `<` reaches EOL with no name. See #jsx-element-multiline.
1156+
{ include: '#jsx-element-multiline' },
11491157
{ include: '#jsx-expression' },
11501158
{ include: '#jsx-entity' },
11511159
],
@@ -1265,6 +1273,72 @@ function generateJsxPatterns(langName: string, identRegex: string, jsx: JsxInfo,
12651273
],
12661274
};
12671275

1276+
// ── jsx-element-multiline: `<` alone at EOL, tag NAME on a later line (#825) ──
1277+
// The single-line tag-open begins (`(<)\s*name`) can't span the `<`/name line
1278+
// break because a TextMate `begin` regex is matched within ONE tokenizeLine, so
1279+
// its `\s*` never crosses a newline. This region decouples the two: it opens on a
1280+
// lone `<` (one whose only same-line tail is whitespace — `(<)(?=\s*$)`), scoping
1281+
// it `punctuation.definition.tag.begin`, then its INNER patterns — applied
1282+
// per-line while the region stays open — pick up the name (with leading
1283+
// whitespace) when it arrives on a following line, exactly like a single-line
1284+
// open tag's attributes are matched line by line.
1285+
//
1286+
// Children-only and unambiguous: in JSX children a bare `<` is ALWAYS a tag
1287+
// opener (tsc rejects a stray `<` in children — `<div>a < b</div>` is a parse
1288+
// error), so a lone `<` reaching EOL here can only be a split tag-open. The
1289+
// `(?=\s*$)` guard means it fires ONLY when no name follows on the same line, so
1290+
// a normal `<span …>` / self-closing `<br/>` / close `</span>` on one line keep
1291+
// their existing (single-line) tokenization untouched. It lives only inside
1292+
// `#jsx-children`, never at expression-start, so a split comparison `a <\n b` or
1293+
// generic `f<\n T>` outside JSX is never reached.
1294+
//
1295+
// Handles BOTH shapes after the split: a self-closing `<\n name … />` ends at the
1296+
// `/>` (first `end` alternative), and an open `<\n name …> … </name>` runs through
1297+
// an inner open-tag-body (name + type-args + attributes up to `>`) into a children
1298+
// region and ends at the matching `</name>` (second `end` alternative).
1299+
result['jsx-element-multiline'] = {
1300+
name: `meta.tag.${langName}`,
1301+
begin: '(<)(?=\\s*$)',
1302+
beginCaptures: { '1': { name: tagBegin } },
1303+
// End on EITHER the self-close `/>` (self-closing split tag) OR the `</name>`
1304+
// close (open split tag). The self-close alt is first so `/>` wins over a stray
1305+
// `>` interpretation; its capture group is 1, the close-tag groups are 2 (`</`),
1306+
// 3..7 (name sub-captures), 8 (`>`).
1307+
end: `(?:(${escapeRegex(jsx.selfCloseTok)})|(${escapeRegex(jsx.closeTok)})\\s*(?:${nameRe})?\\s*(>))`,
1308+
endCaptures: {
1309+
'1': { name: tagEnd },
1310+
'2': { name: tagBegin },
1311+
...nameCaptures(3), // name sub-captures 3..7 (ns, sep, member, intrinsic, component)
1312+
'8': { name: tagEnd },
1313+
},
1314+
patterns: [
1315+
// children region — entered the moment the open tag's `>` has been seen, and
1316+
// listed FIRST so its zero-width `(?<=>)` begin wins at the `>` boundary over
1317+
// the open-tag-body's name match below. This is what stops the name pattern
1318+
// from re-firing on a child word (e.g. `txt` in `<\n span>txt</span>`): once
1319+
// `>` opens this region, matching is confined to `#jsx-children` (which does
1320+
// NOT include the open-tag body), so a bare child word stays `meta.jsx.children`
1321+
// instead of being mis-scoped as a second tag name. Before any `>` this begin
1322+
// can't match (no preceding `>`), so the open-tag body picks up the name first.
1323+
{
1324+
begin: '(?<=>)',
1325+
end: `(?=${escapeRegex(jsx.closeTok)})`,
1326+
contentName: `meta.jsx.children.${langName}`,
1327+
patterns: [{ include: '#jsx-children' }],
1328+
},
1329+
// open-tag body: the NAME (on a later line, leading ws consumed) then
1330+
// type-args / attributes, up to — but not consuming — the `>` or `/>` (the
1331+
// outer `end` closes a self-close `/>`; a plain `>` opens the children above).
1332+
{
1333+
begin: `\\s*${nameRe}`,
1334+
beginCaptures: nameCaptures(1),
1335+
end: `(>)|(?=${escapeRegex(jsx.selfCloseTok)})`,
1336+
endCaptures: { '1': { name: tagEnd } },
1337+
patterns: [...tagTypeArgsInclude, { include: '#jsx-attributes' }],
1338+
},
1339+
],
1340+
};
1341+
12681342
// ── jsx-fragment: `<> children </>` ──
12691343
result['jsx-fragment'] = {
12701344
name: `meta.tag.${langName}`,

‎test/tsx-issue-cases.ts‎

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,11 @@
99
// into one component token. The derived grammar disambiguates both. (#1033 — a JSX component
1010
// with a generic type argument — the official already handles; Monogram now matches it.)
1111
//
12-
// The ledger is HONEST, not cherry-picked: it also keeps cases the derived grammar ties or loses —
13-
// #825 is a both-fail (a `<` and the tag name split across lines defeats both line-oriented
14-
// grammars, the one remaining only/no-one-solves case). The rest the derived grammar now wins or
15-
// ties: #794 / #754 / #585 / #667 / #624 / #1033 are both-pass — a non-null `!` then `/` division
12+
// The ledger is HONEST, not cherry-picked. #825 — a `<` and the tag name split across lines —
13+
// defeats the official (a TextMate `begin` is single-line, so its tag-open can't span the break);
14+
// the derived grammar recovers it with a children-only multi-line tag-open (a lone `<` is an
15+
// unambiguous tag opener inside JSX children), so #825 is now an only-Monogram win. The rest the
16+
// derived grammar wins or ties: #794 / #754 / #585 / #667 / #624 / #1033 are both-pass — a non-null `!` then `/` division
1617
// keeps the `/>` closing the tag; a JSX element after a `/**/` block comment switches into JSX; a
1718
// `//` comment inside an open tag is scoped as a comment; real reported cascades both now handle.
1819
// Each of #794 / #754 / #585 was an only-official miss until the derived grammar caught up. #754 in
@@ -87,17 +88,20 @@ export const cases: Case[] = [
8788
{ id: '#754', title: 'JSX element right after a `/**/` block comment', src: `const a = /**/ <Element />;`,
8889
checks: [{ at: 'Element', want: isTag, desc: 'the post-comment `<Element />` is a JSX tag, not a `<` comparison' }] },
8990

90-
// #825 (BOTH miss this — a PROVEN structural TM limit). `<` and the tag name split across lines —
91+
// #825 (only-Monogram — the official still misses it). `<` and the tag name split across lines —
9192
// a chevron alone on one line, the name on the next. Valid JSX (tsc: nested JsxElements, 0 diags)
92-
// and Monogram's PARSER accepts it; the gap is purely the highlighter. A TextMate `begin` regex is
93-
// single-line, so the tag-open `(<)\s*name` cannot span the `<`/name line break — `\s*` never
94-
// crosses the newline within one `tokenizeLine`. A multi-line construct can't recover it cleanly
95-
// either: a lone `<` is only unambiguously a tag inside children (a bare `<` elsewhere is a split
96-
// comparison `a <\n b` or generic `f<\n T>`), and even in children it would need a bespoke
97-
// multi-line tag-open neither line-oriented grammar implements. Both scope `span` as
98-
// `meta.jsx.children` (plain text). The frontier neither reaches today.
93+
// and Monogram's PARSER accepts it. A TextMate `begin` regex is single-line, so the single-line
94+
// tag-open `(<)\s*name` cannot span the `<`/name line break — `\s*` never crosses the newline
95+
// within one `tokenizeLine`, which is why the official (and Monogram's single-line elements) leave
96+
// `span` as `meta.jsx.children`. The official has no recovery; Monogram does. The lone `<` here is
97+
// in JSX CHILDREN, where a bare `<` is ALWAYS a tag opener (tsc rejects a stray `<` in children —
98+
// `<div>a < b</div>` is a parse error), so it is UNAMBIGUOUS. Monogram adds a children-only
99+
// multi-line tag-open (gen-tm's `jsx-element-multiline`): it opens on a lone `<` reaching EOL
100+
// (`(<)(?=\s*$)`) and recovers the name — plus attributes, type-args, children, and the matching
101+
// close — when they arrive on following lines. Scoped to children (never expression-start), so a
102+
// split comparison `a <\n b` or generic `f<\n T>` outside JSX is untouched. `span` → `entity.name.tag`.
99103
{ id: '#825', title: '`<` and tag name on separate lines', src: `const demo =\n <div>\n <\n span className="foo">\n </span>\n </div>;`,
100-
checks: [{ at: 'span', want: isTag, desc: 'the `span` after a lone `<` is a tag name, not JSX text (both grammars miss this)' }] },
104+
checks: [{ at: 'span', want: isTag, desc: 'the `span` after a lone `<` is a tag name, not JSX text (Monogram recovers the split tag; official misses it)' }] },
101105

102106
// #667 (BOTH solve this now). An arrow function plus a ternary inside a JSX attribute used to
103107
// cascade — the ternary LHS was mis-scoped `meta.parameters` and broke the rest. Both grammars now

0 commit comments

Comments
 (0)