Immediate token handling and "next token" are incompatible

[jordanbtucker]

The spec makes the following two statements:

https://html.spec.whatwg.org/multipage/parsing.html#tokenization

When a token is emitted, it must immediately be handled by the tree construction stage. The tree construction stage can affect the state of the tokenization stage, and can insert additional characters into the stream. (For example, the script element can result in scripts executing and using the dynamic markup insertion APIs to insert characters into the stream being tokenized.)

https://html.spec.whatwg.org/multipage/parsing.html#next-token

The next token is the token that is about to be processed by the tree construction dispatcher (even if the token is subsequently just ignored).

These two statements seem to be incompatible with each other. How can the tree constructor know what the "next token" is if the tokenizer is supposed to wait for the tree constructor to finish its steps?

For example, take the following steps from the in body insertion mode.

When the user agent is to apply the rules for the "in body" insertion mode, the user agent must handle the token as follows:
...
A start tag whose tag name is "textarea"
Run these steps:

1. Insert an HTML element for the token.
2. If the next token is a U+000A LINE FEED (LF) character token, then ignore that token and move on to the next one. (Newlines at the start of textarea elements are ignored as an authoring convenience.)
3. Switch the tokenizer to the RCDATA state.
4. Let the original insertion mode be the current insertion mode.
5. Set the frameset-ok flag to "not ok".
6. Switch the insertion mode to "text".

So, if the tokenizer has just emitted the start tag token, then it is supposed to wait for the tree constructor to run these steps before parsing the next token. How does the tree constructor know whether the next token is a \n character token if the tokenizer hasn't parsed it yet? When "next token" appears, does that mean the tree constructor is giving the tokenizer permission to parse another token?

[domenic]

/cc @whatwg/html-parser in case they can help clarify.

[inikulin]

I don't see any ambiguity here. None of the quotes above state that tokenizer can't emit more tokens by request as previous token is processed by a tree construction stage.

However, the incorrect part is that we are asking for the next token before switching tokenizer to the RCDATA state.

[jordanbtucker]

@inikulin On second look, I don't see a contradiction either. The spec doesn't say that the tokenizer must wait for the tree constructor to finish its steps. That's just how I imaged it looking in code. Thanks for the clarification.

[domenic]

Phew, I'm glad folks figured this out without me having to fully understand the problem :). So it sounds like there is an issue here, which is that from the textarea state excerpt quoted, we should switch steps 2 and 3? Is the problem potentially more general and we should survey other states?

[jordanbtucker]

@domenic Just my two cents, yes it does appear that swapping steps 2 and 3 should resolve the issue.

There are only two places where "next token" is used: after a textarea start tag and after a pre or listing start tag. In both places, it's used to ignore a potential new line. Since pre can contain other elements, it doesn't need to switch to the RCDATA state. So it appears that this issue is isolated to textarea.

[domenic]

Awesome! Would you be interested in making a pull request to correct that?

[jordanbtucker]

Awesome! Would you be interested in making a pull request to correct that?

See #5032

[gsnedders]

On second look, I don't see a contradiction either. The spec doesn't say that the tokenizer must wait for the tree constructor to finish its steps. That's just how I imaged it looking in code. Thanks for the clarification.

I think logically it must, though? When the tree constructor can mutate the tokenizer state it clearly can't operate entirely in parallel. You'd have to know when you can safety resume the tokenizer.

This seems like a bug introduced from when we started skipping LF in textarea and pre and while it's obvious what the intent of the spec is I'm not sure how we can resolve this easily. I think the easiest thing might be getting rid of "next token" entirely and adding a new insertion mode that skips an LF character then switches to text?

[hsivonen]

In Validator.nu/Firefox, there's a flag needToDropLF on the tree builder. It is

• Set to false at the start of the parse.
• Set to false at the beginning of processing a non-character token.
• Set to true when processing a textarea, pre, or listing start tag token. (Happens after the previous bullet point.)
• Set to false after the tree builder has processed the generated pre start tag for text/plain. (After the previous bullet point.)
• Checked (and then always set to false) at the start of processing of character tokens (if the incoming character token is LF and the flag was true, the LF is dropped).

I believe the spec should have this kind of flag.