Thoughts on HTMLish Language Support

[ematipico]

from rome/tools#4077

The HTMLish languages include HTML and HTML-based template languages like: Vue templates, Angular templates, Grale templates, etc.

The goal of this discussion is to share design constraints and ideas.

Objective

Enable code reuse for lints and formatting for the different HTML languages. For example, implement the logic for accessibility lints once and use it for all HTMLish languages.

The objective is not to optimize binary size but enable code reuse.

Design considerations

HTMLish languages are not additive

What I mean by additive is that a language dialect like TypeScript or JSX adds additional syntax to a language:

• If you use TypeScript syntax in a JavaScript file, it becomes TypeScript.
• If you use JSX syntax, it becomes a JSX file
• If you use JSX and TypeScript, it becomes a TSX file

The same applies to JSON, JSON5, and JSON6: Using any JSON6-only syntax makes your whole file a JSON6 file.

That's because the "extension" dialects are supersets of the language they extend:

• JavaScript ⊆ TypeScript
• JavaScript ⊆ JSX, TypeScript ⊆ TSX, JSX ⊆ TSX (with a very few exceptions)
• JSON ⊆ JSON5 ⊆ JSON6

While it's true that you get an Angular template if you use any angular syntax in HTML, it doesn't work if you start mixing template dialects: What format does a file use if you use an Angular expression inside a Vue template? Vular, Angulue?

We'll run into the same problem with JavaScript if we add support for another JavaScript dialect, e.g., Flow. You could even argue that it's a situation we have today with SvelteJs, except that SvelteJs doesn't extend the JavaScript syntax but changes the language semantics.

Naming

HTML element and attribute names are not case insensitive but are case sensitive in Angular (at least when using TypeScript), JSX, and VueJS templates --- and potentially others.

Furthermore, some template languages support multiple syntaxes to assign an attribute value:

• attribute syntax <a href="./link">body<a>
• binding syntax: Vue.js <a v-bind:href="link">body</a>

That means it is necessary for some languages to check for static attributes and bindings when testing if an attribute is set.

JSX

JSX is defined as part of the JavaScript language but otherwise fits the bill of an HTML template language. Meaning reusing code for HTMLish lints rule for JSX is desired but doing so ultimately means that either:

• JSX must be extracted from JavaScript into the HTMLish language (unclear how this would work considering the tight coupling to the JS parser)
• The solution we seek must work across different syntax languages

Design Ideas

Single HTMLish language

Create a unified HTMLish grammar that unifies all different HTMLish dialects.

AnyHtmlishAttribute = VueBinding| AngualarBinding | HtmlAttribute | ... 

This does not mean that all dialects use the same parser. Language could either share the parser implementation, or each comes with its own.

This approach does not allow for code reuse between JSX and HTMLish

Unified HTMLish syntax

The idea is to have language-specific grammars with a unifying language on top.

AnyHtmlishRoot = AngularRoot | VueRoot | HtmlRoot | ... 
AnyHtmlishAttribute = AnyAngularAttribute | AnyVueAttribute | ...

This approach is very compelling at first but has two shortcomings

• The union nodes cannot implement AstNode because it's unclear what to use for the L: Language type parameter (the nodes are from many different languages)
• Much of our infrastructure is typed over L: Language.

HTMLish traits

Each HTMLish language has its own grammar and traits abstract the common operations used by the linter that allow iterating elements, finding attributes, testing attribute values, etc. Lints would then either be generic over the HTMLish trait implementation or a trait with methods to perform some language-specific operations.

This trait system may be supplemented by an HTMLish semantic model that allows fast queries for specific elements/attributes.

It's not entirely clear to me how this would work on the formatting side, but I could envision that the formatting of an Element is a trait that can then be implemented by language-specific formatters.

[denbezrukov]

I've been following this discussion on designing a flexible and reusable system for HTMLish languages, and I'd like to share some thoughts that might contribute to our collective understanding and approach.

Given the complexity and diversity of HTMLish languages (including HTML, Vue, Angular, etc.), I believe that employing separate parsers for each language, coupled with a trait-based system for linters and formatters, presents a viable and robust solution. This approach allows for nuanced parsing that can accurately reflect the unique syntax and semantics of each language while providing a unified interface for tooling, thereby facilitating code reuse and extensibility.

Separate Parsers for Each Language

Creating dedicated parsers for each HTMLish language ensures that the parsing process is tailored to the specific needs and features of each language. This specificity is crucial for accurately capturing the nuances of each language, including unique attributes, directives, and template syntaxes. It also allows for performance optimizations specific to the parsing and analysis needs of each language, enhancing efficiency and effectiveness.

Trait-Based Linters/Formatters

A trait-based approach for linters and formatters can significantly streamline the development of these tools. By defining common operations, checks, and analyses in traits, we can apply these tools across different languages with minimal code duplication. This not only speeds up the development process but also ensures consistency in how different languages are linted and formatted, providing a more uniform developer experience.

Extending Codegen for Grammar Reuse

One of the challenges in this approach is the potential duplication of effort in defining grammar for similar constructs across languages. To address this, I propose extending our code generation system to allow for the sharing of grammar components among the different language parsers. This would involve:

• Identifying Common Grammar Elements: We start by identifying the elements that are common across HTMLish languages, such as basic HTML tags, common attributes, and template syntax elements that are shared among languages.
• Creating a Shared Grammar Library: These common elements would then be abstracted into a shared library that can be included in the grammar definitions of each language. This allows us to define these elements once and reuse them across different parsers.
• Allowing Language-Specific Extensions: Each language-specific grammar can extend the shared definitions with its unique elements and rules. This ensures that while we maximize code reuse, we also retain the flexibility to accurately capture the unique features of each language.

This approach not only fosters code reuse but also ensures that our parsers remain flexible and accurate, capable of adapting to the specific requirements of each HTMLish language.

Conclusion

By combining separate parsers with a trait-based approach for tooling and an extended code generation system for grammar reuse, we can achieve a balance that allows for accurate, language-specific parsing and efficient, reusable tooling. This strategy respects the unique aspects of each language while promoting a more unified and efficient development process for linters, formatters, and other analysis tools.

I'm eager to hear your thoughts on this approach and how we might refine it further to address our collective goals.

[ematipico]

I also think that a trait system is the best choice in the long run.

The initial set up will be quite difficult and awkward, because we will need to modify our code gen just for this syntax .

Maybe, instead of starting from the grammar, we should start from an initial implementation.

Perhaps we could start by implementing a trait for the attributes, and then trying to make it "specific" for HTML and another language.

What do you think about it?

[Conaclos]

Just wondering: are there so many differences between htmlish languages that justify separate grammar and CST? We could use a common CST and a parser mode to parse accurately language?

[ematipico]

As explained in the beginning:

The union nodes cannot implement AstNode because it's unclear what to use for the L: Language type parameter (the nodes are from many different languages)

Much of our infrastructure is typed over L: Language.

[Conaclos]

If we use a single grammar, we have a single language. I don't understand your comment @ematipico .

[ematipico]

If we use a single grammar, we have a single language. I don't understand your comment @ematipico .

We can't re-use logic and nodes. One of the objectives would be to add JSX to the HTML dialects.

For example, Astro and Vue allow JSX syntax in their files. Astro uses a dialect of JSX. Although, today JSX nodes are typed over the JavaScript Language, so it's unclear how we could re-use what we have inside new files.

[conartist6]

I went a rather different way on this one: language embedding all the way down. I see the TSX language as TSX syntax embedded in TS syntax embedded in JS syntax embedded in comment syntax embedded in whitespace.

[jpike88]

I believe this approach shouldn't require 'rust for everything' as an approach out the gate. Parsing angular source code is a non-trivial task, and projects like https://github.com/prettier/angular-estree-parser exist that take that problem out of the equation. Why not call that somehow to get your syntax tree, pass it back into rust and then apply the transformations?

Maybe use https://docs.rs/neon/latest/neon/index.html for that.

Then, over time, when time permit, replace the AST syntax parser with a rust implementation.

For angular:

Template code is either in .component.html, or in in the template property of the @component decorator. When that is detected, activate a dedicated parsing pipeline.

Make a call

[dyc3]

Vue

I've made a very rough state diagram for Vue's HTMLish template syntax. Ref: https://vuejs.org/guide/essentials/template-syntax.html

stateDiagram-v2
    [*] --> HtmlTag: <
    HtmlTag --> HtmlText: >
    HtmlText --> AnyJsExpression: {{
    AnyJsExpression --> HtmlText: }}
    HtmlText --> HtmlTagClose: </
    HtmlTagClose --> [*]: >

    HtmlTag --> HtmlAttributeList: (space)
    HtmlAttributeList --> HtmlText:>

    state HtmlAttributeList {
        state attr_type <<choice>>
        state vue_attr_eq <<join>>

        [*] --> attr_type
        attr_type --> VueDirectiveName: v-
        attr_type --> VueVBindShorthand: colon
        attr_type --> VueVOnShorthand: @
        attr_type --> HtmlAttribute: else

        VueDirectiveName --> VueDirectiveArgument: colon
        VueVBindShorthand --> VueDirectiveArgument
        VueDirectiveArgument --> VueDirectiveArgumentDynamic: [
        VueDirectiveArgumentDynamic --> vue_attr_eq: ]=
        VueVOnShorthand --> VueDirectiveArgument
        VueDirectiveArgument --> VueDirectiveModifier: .
        VueDirectiveArgumentDynamic --> VueDirectiveModifier: ].
        VueDirectiveModifier --> vue_attr_eq: =
        VueDirectiveArgument --> vue_attr_eq: =

        vue_attr_eq --> VueDirectiveValue: "
        VueDirectiveValue --> [*]: "
    }

Loading

A couple of notes:

• This doesn't really account for self closing elements
• VueDirectiveValue would be AnyJsExpression, as long as it doesn't contain a ", which would result in a syntax error.
• Doesn't account for directives with no value, eg. v-else

But this gives me the impression that writing a grammar for Vue wouldn't be that bad.

Svelte

Svelte's grammar is going to be significantly more complex because of the additional syntax.
Ref:

• https://svelte.dev/docs/basic-markup
• https://svelte.dev/docs/logic-blocks

stateDiagram-v2
    [*] --> HtmlTag: <
    HtmlTag --> HtmlText: >
    HtmlText --> HtmlTagClose: </
    HtmlTagClose --> [*]: >

    HtmlTag --> HtmlAttributeList: (space)
    HtmlAttributeList --> HtmlText:>

    state HtmlAttributeList {
        state attr_type <<choice>>
        
        [*] --> attr_type
        attr_type --> SvelteBindingShorthand: {
        attr_type --> HtmlAttributeName: A-z
        HtmlAttributeName --> HtmlAttributeValue: =
        HtmlAttributeValue --> HtmlAttributeText: "
        HtmlAttributeValue --> SvelteBindingValue: {
        HtmlAttributeValue --> SvelteBindingValue: "{

        SvelteBindingValue --> [*]: }
        SvelteBindingValue --> [*]: }"
        HtmlAttributeText --> [*]: "

        SvelteBindingValue --> SvelteSpreadBindingShorthand: ...
        SvelteSpreadBindingShorthand --> [*]: }

        SvelteBindingShorthand --> SvelteBindingValue
    }

    HtmlText --> SvelteLogicBlockStart: {#
    SvelteLogicBlockStart --> SvelteLogicBlockKeyword
    note right of SvelteLogicBlockKeyword
    Obviously, after this would be expressions
    and some other tokens, but i'm not 
    diagramming it for now
    end note
    SvelteLogicBlockKeyword --> HtmlText: }

    HtmlText --> SvelteLogicBlockMiddle: {(colon)
    SvelteLogicBlockMiddle --> SvelteLogicBlockKeyword
    note right of SvelteLogicBlockMiddle
    Not sure what this is actually called,
    too lazy to look it up right now
    end note

    HtmlText --> SvelteLogicBlockEnd: {/
    SvelteLogicBlockEnd --> SvelteLogicBlockKeyword

Loading

• Doesn't account for let: 2 way bindings
• Doesn't fully diagram the logic blocks syntax for brevity
• Doesn't account for the change to js behavior via the $: syntax
• Doesn't account for special tags: https://svelte.dev/docs/special-tags
• Doesn't account for element directives: https://svelte.dev/docs/element-directives
• Doesn't account for special elements: https://svelte.dev/docs/special-elements

Conclusion

Based on this little exploration, I don't think we are going to have any many shared tokens between these template languages, with the sole exception being anything that is already valid html.

Edit: Took a brief look at Angular, and it seems to share the {{ }} tokens and it looks like they serve the same purpose.

[dyc3]

I've opened a PR attempting to add Vue's template syntax to the HTML grammar: #3369

[dyc3]

There's one problem I hadn't considered before. We currently treat .vue/.svelte/others as JS files.

biome/crates/biome_js_syntax/src/file_source.rs

Lines 196 to 209 in 40727ca

	/// Astro file definition
	pub fn astro() -> Self {
	Self::ts().with_embedding_kind(EmbeddingKind::Astro)
	}
	/// Vue file definition
	pub fn vue() -> Self {
	Self::js_module().with_embedding_kind(EmbeddingKind::Vue)
	}
	/// Svelte file definition
	pub fn svelte() -> Self {
	Self::js_module().with_embedding_kind(EmbeddingKind::Svelte)
	}

This is so that we are able to format and lint JS within blocks of code that we know is JS, (eg. vue and svelte's <script> tags), but it's kinda hacky.

I think that ultimately having separate grammars for each html-ish language is going to be inevitable. From a DX perspective, it would kinda make sense for all the vue lint rules to go in biome_vue_analyze, not biome_html_analyze.

However, it's unclear to me how we would switch languages mid parsing. I'm not aware of any place we are currently doing that. We need to be able to in order to correctly parse VueDirectiveValue and VueTemplateInterpolation (as named in #3369), for example.

[ematipico]

I'm not aware of any place we are currently doing that

Nope, that will be the first time, and that's what we want to figure out.

HTML will be the only language where we will have to switch the parsing midway, but I don't think that's an issue, I believe it's fine if the html parser (the crate) depends on the JS one and the CSS one.

[jpike88]

Angular's control flow syntax sets it apart in a way that isn't what I'd call HTML-ish, I think it makes sense to treat each language as a seperate problem to solve. This also allows for the possibility of using existing AST parsing libraries as a intermediary step if the situation calls for it

[Sec-ant]

I have some unpolished thoughts on the js/css parser reusing in html-ish languages: #3334 (comment). Bascially, I think we can evaluate the idea of considering them as embedded languages. We can have some default out-of-box internal rules to support <script>, <style>, and even <script type="application/ld+json"> and <script type="importmap">. The benefit is that with the embedded language + plugin infra, it will be trivial for us to compose parsers to support new syntaxes in the future, it will also be easier for users to customize the behavior, for example: vue custom blocks

[Zooce]

These observations for Vue and Svelte are spot on. I just wanted to mention Angular as well - like with its new control flow syntax https://angular.dev/guide/templates/control-flow and old control flow syntax (<div *ngIf="show"></div>) and input/output binding (<component [someInput]="someSignal()" [(twoWay)]="data" (output)="doThis($event)" (click)="onClick()"></component>).

[junaga]

self closing tags

I'd like to very quickly and shortly mention the issue with / in <div /> as jakearchibald.com/2023/against-self-closing-tags-in-html said, just in case someone missed that.

Jake said it already very very well. But basically he said:

JSX is not HTML. XHTML is not HTML. HTML is HTML. github:whatwg/html is HTML. In HTML the / is ignored. It's boilerplate. - notarealquote

others agree

That said, i think Biome should trim / from all elements in HTMLish languages. Perhaps with an "formatter" option to "off" or "preserve" this in addition to "on".

[ciriousjoker]

Not sure if this is relevant, but a formatting option that automatically forces the self closing syntax if there is no content would be nice.

Example:
<div class="abc"></div>
becomes
<div class="abc" />

[jpike88]

@ciriousjoker that seems like a huge readability issue and unconventional for standard HTML practices... if one of my devs starting self-closing div tags I'd be pretty annoyed about it. Only certain tags (like img, br) are supposed to be self-closing.

[dyc3]

Just an FYI, and for anybody that comes across this via search, we just resolved this in #4490.

The legacy reason for "self closing" syntax to be a thing at all is because (probably, this is my speculation) prettier uses a fork of angular-html-parser for it's html parser, which is not spec compliant.

In HTML, void elements are elements that have no children, and therefore have no closing tag, and therefore are self closing. Good examples of void elements are <img> and <br>. And it is correct that the HTML spec considers the self closing syntax <foo /> to be invalid syntax. Browsers ignore the / so its not really a problem that nips devs unexpectedly. This is because HTML defines void elements by their tag name, not the syntax they use. (that's the short version of it)

We still haven't entirely figured out what we are doing for dealing with embedded languages (see #3334 ). Several HTML template languages consider this syntax <foo /> be the determination of a self closing tag. In order to be a good tool for vue, svelte and other users, we have to fully support all the features those framework specific template languages have, which includes the self closing syntax. There will be an option to use prettier's "self closing" syntax for void elements. The default for .html documents will be to use HTML spec compliant syntax for void elements, but in non-.html template ish documents, the default will be self closing syntax.

[ciriousjoker]

@jpike88
I general I agree, but Angular has them, so they need to be supported either way.
Also, I find them to be pretty convenient in many cases. Many components don't take a child and instead just do their thing based on parameters. Adding those closing tags and wasting space just to be html spec compliant (which angular templates aren't anyway) is kind of pointless in my opinion.
Either way is fine I think (always where possible vs never where possible), I just personally prefer never.
Either way, this would require checking which elements are angular components and which are regular browser elements. A whitelist might suffice here though.

[gwennlbh]

fwiw svelte 5 kinda wants to get closer to browser behavior on the whole self-closing thing, see sveltejs/svelte#11052

[pumano]

https://github.com/prettier/angular-html-parser that is used by prettier for html angular

If biome devs can port it to rust, it would be perfect, also it helps to port angular template specific eslint rules

[touhidrahman]

1. Prettier natively supports the new Angular syntax in HTML files. So if Biome is going to be 100% prettier-compatible, then at some point it needs to be done.
2. Biome is extending support for frameworks like Svelte and Vue. Angular as a framework still holds a very good position in the ecosystem. It will be unfair to leave a large number of developers out from using biome.

[edlessz]

Maybe this would be better suited as a plugin? A lot of the heavy lifting for the formatting can be done with GritQL.

[yeoffrey]

Just wanted offer some Svelte & SvelteKit specific thoughts here. Biome for Svelte would be awesome since there aren't many other options besides EsLint and Prettier. They are also quite slow in Svelte projects.

Svelte 5

In some ways, you can think of Svelte 5 as a superset of Svelte 4.

• Svelte 5 still supports the older Svelte 4 syntax, which will have to be included in the parser.
• You now also have the runes API which handles some reactivity now.
• There are now multiple different types of svelte files.
  • .svelte files which are normal.
  • (new) .svelte.js & .svelte.ts files which allow for writing runes in non component files.

SvelteKit

• More files are supported here, like +page.svelte/ts/js, +layout.svelte/ts/js and server-only files that are pure TS.
• This might effect the new multi-file analysis feature that shipped with BiomeJS 2.0.

Svelte's "HTML"

• Nested <style> tags are supported. I'm not sure if this effects the current HTML parser, but I don't think that this is technically valid HTML.
• Custom properties on tags might be a headache to deal with, since they can really be any css.
• Svelte also officially supports self-closing tags for some elements (input for example). As well, components can be self closing too.

Existing linting & formatting of Svelte files

Most projects use Eslint and Prettier for formatting svelte files.

• Eslint uses the eslint-plugin-svelte which internally uses svelte-eslint-parser which is officially supported by the SvelteJS team.
• Prettier uses the prettier-plugin-svelte

[yeoffrey]

Also adding here that I'd like to work on this if the Biome team would like a dedicated Svelte implementation. I'm not familiar with React / Vue / Angular so I cannot speak for those frameworks.

[dyc3]

PRs are certainly welcome! 😄 Here's some things that we can do in the short term:

I don't think we handle .svelte.js/ts files yet. We automatically add svelte runes to the globals configuration for .svelte files, we could do the same thing for .svelte.js/ts files. I've opened #6167 for this, if you are interested.

Nested style elements shouldn't be a problem, ultimately, once #3334 is fully solved. It could be a problem if we added support for css in .svelte files using our current method. The way we currently support js in .svelte files is that we regex for the script tag and extract just the js text and process that, which mostly relies on there only being 1 script tag. You could certainly take a stab at adding support for the style sections.

Regarding linting: There's really no blockers for svelte lint rules that only touch ts/js files. The thing is, most of the useful rules that I can think of require looking at .svelte files. If there's lint rules that can fit within these restrictions, you are welcome to make PRs for them and add a new svelte domain for them to live in.

---

Full support for svelte is ultimately blocked by support for HTML (which is currently experimental), and by embedded language parsing/formatting (#3334). If you're interested in really helping drive this home, I would certainly appreciate the help!

[yeoffrey]

Thanks for the great write up. I'm happy to help drive #3334 home if you'd like the help. I just need to get up to speed on that feature. I've read through the issue but I'm not sure where we stand on the work ATM. Whats the next step there? (We can bring this convo there as well)

In terms of what we can already support, I've listed below all the svelte-eslint rules that we could implement right now.

• svelte/block-lang
• svelte/no-ignored-unsubscribe
• svelte/no-reactive-functions
• svelte/no-reactive-literals
• svelte/no-svelte-internal
• svelte/no-unnecessary-state-wrap
• svelte/prefer-const
• svelte/prefer-writable-derived
• svelte/require-event-dispatcher-types
• svelte/require-stores-init
• svelte/require-event-prefix
• svelte/derived-has-same-inputs-outputs

Others will require the HTML-like parser to be finished.

Also currently the indentation on the script tags is at the beginning of the line, not tabbed or spaced like it currently formats with prettier. Maybe theres an easy win there?

[dyc3]

These look pretty easy to implement:

• svelte/require-stores-init
• svelte/require-event-prefix

I don't think svelte/prefer-const is possible right now.

You're absolutely welcome to take a stab at #3334! (we can keep the discussion of that in there). The problem space is rather complex so it's easy to get lost. At a high level, we're kinda thinking parsing will go something like this: #3334 (comment) Personally, I would recommend trying to implement a couple lint rules first so you're a little more familiar with the project before attempting #3334.

[ematipico]

I opened the following umbrella. I'm excited to share that we started the work for supporting Vue, Svelte and Astro! #6657

It will take some time, and if you're interested in the issue, subscribe and help us :)