Implement multi-language support by msujew · Pull Request #130 · TypeFox/fastbelt

msujew · 2026-07-23T14:52:40Z

Closes #96

Does as the title says, similar to how it is outlined in the issue:

Adds a fastbelt/cmd package adopters can use to write their own CLIs to generate Fastbelt code.
Adds a LanguageSelector service that is used to determine during runtime what language a URI/document belongs to.
Implements a switch-case based selector in the parser and lexer to circumvent the need for a separate service instance per language.
Adds a multilang example that shows that this is working as expected (including parser, lexer and completion tests).

FYI, while this has a Plugins field in the API, plugins aren't actually supported in this PR. This will need a follow-up.

cdietrich · 2026-07-24T07:56:39Z

wondering about the common lexical scope and an option to have a lexer per entry rule
#131

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: `40d792c`	Previous: `d803417`	Ratio
`BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s`	`13.16` MB/s	`5.21` MB/s	`2.53`

This comment was automatically generated by workflow using github-action-benchmark.

msujew · 2026-07-27T14:26:48Z

@cdietrich Thanks for the input. I plan to finish implementing this once #127 lands, since it refactors a lot of the lexer infrastructure.

ssmifi

Looks solid, thanks. I have a couple of questions to get a better understanding before merging and some documentation seems to be stale.

ssmifi · 2026-08-07T16:07:48Z

@@ -55,7 +55,7 @@ func runGenerateCLI(opts generateOptions) error {
 	}

 	sc := grammar.CreateServices()


Nice separation.
Question: The following lines are conceptionally doing for one file what parseAndMerge and reportDiagnostic do generically now. I was wondering if we can de-duplicate some code here and use come common helper?

ssmifi · 2026-08-10T09:09:23Z

+
+type DocumentMatcher func(uri URI) bool
+
+// DocumentFilter matches a document by LSP language id and/or a glob over the


DocumentSelector?

ssmifi · 2026-08-10T09:13:04Z

+	"typefox.dev/fastbelt/util/service"
+)
+
+type DocumentMatcher func(uri URI) bool


As this is an exported symbol, it should probably get a doc, even if self-explanatory.

Lotes · 2026-08-10T14:58:52Z

+	service.Put[core.LanguageSelector](
+		sc,
+		core.NewDefaultLanguageSelector(sc,
+			core.NewDocumentSelectorWithPatterns("arithmetics", "**/*.arithmetics"),


Puh, a document selector can also influence the folders up the file hierarchy. Also here you seem to be forced to use „**/“… this is broader than an array of file extensions. Not sure if we should go for this. You always need to prepend „**/*“ to your „.ext“.
Is this case-insensitive btw?

Also here you seem to be forced to use „**/“… this is broader than an array of file extensions.

You're not forced to do it. Adopters can implement their own FileExtensionDocumentSelector if they want to. If absolutely required, we can also implement this in the framework, but I'm not really convinced that this is better than the glob pattern.

I'm not happy how this works in Langium, because the idea to use file extensions as the defining identifying feature of languages has a large negative impact on the API design.

Is this case-insensitive btw?

No, but neither is Langium's file extension comparison AFAIK.

OK, I am not really for or against it. It just feels unusual.
But it serves the same purpose as before and goes even beyond that, which might be a good sign to keep it.

Lotes · 2026-08-14T09:02:32Z

+// documents.
+type Language struct {
+	Entry      string
+	LanguageID string


Add a lexer mode, after #127 is merged.

Or combine it within the syntax of the entry rule:

entry XRule on mode YMode: ...

Omitting on mode YMode falls back to the default mode.

Lotes · 2026-08-14T09:05:26Z

@@ -0,0 +1,15 @@
+grammar MultilangModel


I thought this PR is also about multi file support.
But yes, this might be a smaller step towards this direction.

Lotes · 2026-08-14T09:12:50Z

+	DocStateLocalSymbols // 0x0010
 	// DocStateLinked marks that cross-references were linked.
-	DocStateLinked // 0x0010
+	DocStateLinked // 0x0020


Why the comment "//0x0001" ...? Who needs it?
The only reason in my eyes: to not forget the flag nature of this constant group.

Alternative: Rename the constants to reflect the flag nature.

DocStateFlagLinked

DocStateBitMaskLinked

msujew added 2 commits July 27, 2026 15:59

Implement multi-language support

03c238d

Rebase

9f6d652

msujew force-pushed the msujew/multi-language-support branch from 78988e1 to 9f6d652 Compare July 27, 2026 14:14

github-actions Bot reviewed Jul 27, 2026

View reviewed changes

Fix scaffolding

18be4d2

msujew added 2 commits July 27, 2026 17:14

Remove DocumentParser service

25a7226

Support multi-language lexers

292cbfb

msujew mentioned this pull request Aug 3, 2026

Add per-language lexers to eliminate cross-language keyword pollution #131

Closed

Add completion tests

227ab88

msujew marked this pull request as ready for review August 5, 2026 14:00

msujew mentioned this pull request Aug 6, 2026

[LSP] Listen to file system events #118

Open

ssmifi reviewed Aug 10, 2026

View reviewed changes

Review comments

40d792c

Lotes reviewed Aug 10, 2026

View reviewed changes

msujew mentioned this pull request Aug 11, 2026

[CLI] Support type lowering in the generated symbol containers #145

Open

Lotes reviewed Aug 14, 2026

View reviewed changes

		@@ -55,7 +55,7 @@ func runGenerateCLI(opts generateOptions) error {
		}

		sc := grammar.CreateServices()


		type DocumentMatcher func(uri URI) bool

		// DocumentFilter matches a document by LSP language id and/or a glob over the

Uh oh!

Uh oh!

Conversation

msujew commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdietrich commented Jul 24, 2026

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

msujew commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssmifi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lotes Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

msujew commented Jul 23, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

msujew commented Jul 27, 2026 •

edited

Loading

Lotes Aug 10, 2026 •

edited

Loading