"in foreign content" is missing from list of insertion modes

[zcorpan]

https://html.spec.whatwg.org/multipage/parsing.html#the-insertion-mode

Initially, the insertion mode is "initial". It can change to "before html", "before head", "in head", "in head noscript", "after head", "in body", "text", "in table", "in table text", "in caption", "in column group", "in table body", "in row", "in cell", "in select", "in select in table", "in template", "after body", "in frameset", "after frameset", "after after body", and "after after frameset" during the course of the parsing, as described in the tree construction stage. The insertion mode affects how tokens are processed and whether CDATA sections are supported.

https://html.spec.whatwg.org/multipage/parsing.html#parsing-main-inforeign is not mentioned, although it is where CDATA sections are supported. If in foreign content is not an insertion mode, then the insertion mode doesn't affect whether CDATA sections are supported...

[annevk]

I looked into this as part of https://bugs.webkit.org/show_bug.cgi?id=189431.

1. I don't think it's an insertion mode. The note is incorrect. Instead it's a set of rules that https://html.spec.whatwg.org/multipage/parsing.html#tree-construction-dispatcher uses, but the underlying insertion mode remains unchanged. E.g., typically it will be "in body".
2. Whether a CDATA token is supported hinges on this:

If there is an adjusted current node and it is not an element in the HTML namespace, then switch to the CDATA section state.

Now as per the WebKit bug, Chromium and WebKit also require the parser to not be in an integration point.

Now the interesting test is the combination of CDATA and U+0000 in this state.

http://software.hixie.ch/utilities/js/live-dom-viewer/?%3Cscript%3Edocument.write(%22%3Csvg%3E%3Ctitle%3E%3C!%5BCDATA%5B%5Cu0000fdsf%5D%5D%3E%22)%3B%3C%2Fscript%3E

Gecko again appears to follow the specification and not emit U+0000 as is required by "in body", but is that what we want? We should probably treat U+0000 explicitly when tokenizing CDATA to ensure it does not get lost.

cc @whatwg/html-parser

[hsivonen]

"in foreign content" indeed isn't an insertion mode.

My understanding of the reason why CDATA sections are allowed as children of HTML integration points is that SVG desc and title as well as MathML annotation-xml could legitimately contain non-HTML, so supporting CDATA sections maximizes compatibility with copypaste from XML.

[annevk]

What do you think about the U+0000 case?

[annevk]

@sirreal since you have been looking at this, do you happen to know if we have test coverage for U+0000?

[sirreal]

My reading is that all of the insertion modes are HTML insertion modes. The tree construction dispatcher dispatches tokens either to an insertion mode for HTML content or to the rules for parsing tokens in foreign content. Insertion modes and the rules for parsing tokens in foreign content are separate.

That distinction seems arbitrary; "foreign content" seems like another insertion mode on first read. However, making "foreign content" another insertion mode would have significant implications. Some examples (not exhaustive):

• parsing elements that contain only text captures the original insertion mode, which is later restored.
• The rules for parsing tokens in foreign content make use of the current insertion mode.
• The reset the insertion mode appropriately algorithm does not consider foreign content.

If foreign content were treated as another insertion mode, there would be much more accounting to manage or much heavier reliance on resetting the insertion mode.

---

Regarding the insertion modes, this line is wrong where it mentions CDATA sections:

The insertion mode affects how tokens are processed and whether CDATA sections are supported.

I opened #12658 to fix it.

---

Regarding the U+0000 NULL handling, there is coverage in WPT primarily via plain-text-unsafe. Null in HTML, foreign content, and at integration points seems to have general coverage.

Two notable coverage gaps:

• Handling of NULL in MathML foreign content. I only see coverage in SVG foreign content and in MathML at integration points. I don't see tests like <math>␀ -> <math>�.
• Fragment tests: I haven't noticed tests specifically for fragment parsing with ␀ at different context nodes.

Here's some of the relevant coverage in wpt:

• <body>␀
• <svg><![CDATA[␀filler␀text␀]]>
• <body><svg><foreignObject>␀filler␀text
• <svg>␀filler␀text

There also appear to be a number of similar tests for integration points in math.

(GitHub's web renderer gets confused somewhere in this document; line numbers may be inaccurate either in reality or the rendered view on GitHub)

[sirreal]

In addition to fixing the inaccuracy with #12658, it would be useful to add a clarification that "foreign content" is not an insertion mode but and adjacent token parsing mechanism.

[annevk]

web-platform-tests/wpt#61370 should tackle the missing test coverage, but it's annoying to review due to U+0000. It suggests WebKit (and Chromium's copy) have a further divergence from the specification.

[annevk]

Also created #12700 for the note. I think that should conclude this issue.