Rethink internal HTML lexer

[ematipico]

There two things to think about

Lex context

At the moment, our HTML parser has a context called HtmlLexContext::TextExpression(TextExpressionKind::Single).

This lex context is directly mapped from the FileSource::svelte. Are there other languages that use { } inside HTML files? If not, should we rename single text expressions to svelte?

How to lex tokens

Svelte introduces some complexities in the HTML parser. Some of them is the introduction of keywords such as if, else, etc.

At the moment, our HTML parser has few keywords, but they are tight to the lexing context. This isn't maintainable anymore because the context can't be used when doing lookup operations.

For example, the following two block require a lookup of the if word:

{:else if expression}
{:else}

The only way to do lookup is with p.nth or p.nth_at, which don't use any context. Nowadays, the parser will return a HTML_LITERAL token. We can't check for cur_text too.

I think we have make those keywords part of the classic consume_token, and we should adapt the HTML parser to get its way around those keywords.

[dyc3]

For this, I was thinking we could just add new punctuation for {#, {:, and {/ which would indicate svelte syntax.

But yeah, the lexer should probably be redone. It's the part I like touching the least.

[ematipico]

But yeah, the lexer should probably be redone. It's the part I like touching the least.

You and I both! I will do this work as part of the implementation of a new feature, because I believe that's where we will see the results of the new lexer.

I had started with the else/if control-flow, but I realised it has too many changes. So, I will implement {@debug ...} instead, which is reasonably straightforward.