Introduce token modes by Lotes · Pull Request #127 · TypeFox/fastbelt

Lotes · 2026-07-10T13:36:05Z

Closes #50

added explicit token modes (lexer modes)
if none is given, one is create implicitly
if not given, it collects all keywords, token declarations and token groups from the grammar
token modes can have following members
- own token declarations
- token refs to top-level token declaration
- own token groups
- keywords
- keyword selection using a RegExp
token commands were also introduced
- push(name) to push a token mode onto the token mode stack
- pop to pop the topmost token mode
- mode(name) to set the topmost token mode
I also extended token declarations to be keywords as well
a lot of changes happened in the lexer generator
- I tidied up the code organization a lot, I hope you like it as well!
- token types are split from group type and token commands
  - it made sense during the journey in order to write a command/type differently in different modes... we can discuss if it still makes sense
- "keywords" I renamed conceptually to KeywordSelector
validations around token commands were created
a lot of tests were added by AI, they caught bugs and show the expectations when using token modes

msujew

Nice! Can you resolve the merge conflict so we can check the performance in the GitHub CI? Locally, it seems to look good. Only a 2-3% loss on the statemachine example, which is acceptable.

Lotes · 2026-07-17T13:44:03Z

@msujew I discovered a problem with my main grammar.

The problem arises when I create a token mode and start to create token declarations within this token mode. As soon as I complete the word "token", the language server seems to end up in a endless loop. It does not react anymore. After completing the declaration and restarting the LS, it works. So the problem is during editing incomplete grammars :-/...

token mode default {
  toke //complete to "token" to see the problem
}

The rule TokenModeMember: TokenDeclUsage | TokenGroupUsage | TokenUsage | KeywordUsage | KeywordSelector is actually having some alternatives with a common (nested) prefix. I hoped that the LL(STAR) is handling this ^^*. Or maybe it is a different problem?

If I would refactor the grammar to have no common prefix in the subrules of TokenModeMember the problem might disappear and the performance would increase.
But it would also make the grammar more difficult I think.

Before I refactor it would be good to discuss whether these members have a good syntax or whether we should alternate it (which could remove the common prefix automatically).

Lotes · 2026-07-17T13:47:27Z

Ideas...

Some quick fixes or code actions come into my mind:

inline a token rule that points to a keyword
and the opposite: extract all keywords to a common token rule

Or

a hover tooltip over a keyword selector which keywords are included

msujew · 2026-07-27T14:38:32Z

I discovered a problem with my main grammar.

Does this require some setup? I could not reproduce this:

But it would also make the grammar more difficult I think.

I would keep it as it is for now 👍

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.50.

Benchmark suite	Current: `1ee2bcd`	Previous: `d803417`	Ratio
`BenchmarkWorkspaceCycle (typefox.dev/fastbelt/examples/statemachine) - MB/s`	`11.92` MB/s	`5.21` MB/s	`2.29`

This comment was automatically generated by workflow using github-action-benchmark.

Lotes · 2026-08-06T08:25:22Z

I discovered a problem with my main grammar.

Does this require some setup? I could not reproduce this:

No, it should fail out of the box. I can also complete this line as you started. But afterwards toke gets underlined and the LS is frozen. I cannot jump to definition or anything else.
I could not find out how to debug this yet.

EDIT:

I found out how to debug. Some kind of endless-loop in entered in this case.

EDIT:

Fixed the endless loop and another issue with the help of @msujew

Lotes

Added some findings and also questions from my side.

msujew

Really solid contribution. Thanks a lot!

I noticed another few missing validations that we should add as part of this PR:

Duplicate keyword/token definitions in the same token mode should result in an error. I.e. you can currently write token mode default { "x" "x" }.
Similarly Token references should be unique in the same token mode. Right know, you can write something like token mode default { hidden WS comment WS } without a validation error appearing.
When a token is declared, but not referenced in any token mode, the diagnostic appears on the first usage site. I would expect it on the token declaration itself. Also, similar to the keyword issue, this should be an error.
A non-default token mode without a pop command should show a diagnostic.

# Conflicts: # internal/grammar/types_gen.go # Conflicts: # internal/grammar/types_gen.go

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io> # Conflicts: # lexer/lexer.go

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

msujew

Looking closer at some of the logic, I found some issues with some specific constellations. Works much better in general now, though!

msujew · 2026-08-19T11:39:13Z

 hidden token WS: /\s+/
 comment token ML_COMMENT: /\/\*[\s\S]*?\*\//
-hidden token SL_COMMENT: /\/\/[^\n\r]*/
+hidden token SL_COMMENT: /\/\/[^\n\r]*/


Formatting: You've removed the EOL here.

msujew · 2026-08-19T12:31:36Z

+			case grammar.TokenGroupUsage:
+				tokenGroup := member.Group()
+				tokenIndex := result.TokenIndex.ByTokenGroup[tokenGroup]
+				pushTokenTypeUsage(tokenIndex, tokenGroup.Modifier(), tokenGroup.Command())


Suggestion: Token groups in token modes show some buggy behavior. Let's take the following grammar code:

token group X { "x" } token mode default { "x" -> push(Y) } token mode Y { X -> pop }

Giving a grammar like this the input xx actually results in an error at the second x, because the X token group does not have start char hints (or is lexible at all). Using X inside a token mode, should actually add its "sub-tokens" to the token mode.

msujew · 2026-08-19T12:41:23Z

+	//if we are stuck on the same token, consume it and try to move on
+	if parserState.lastErrorIndex == parserState.Index {
+		parserState.Index++
+	}


Suggestion: This should not bump the Index beyond parserState.Length. Otherwise we run into issues at EOF.

msujew · 2026-08-19T12:45:18Z

+		}
+
+		for _, token := range tokens.TopLevel {
+			if _, ok := token.Content().(grammar.RegexpTokenElement); ok {


Suggestion: This drops token usage on tokens like hidden token Test: "test".

msujew · 2026-08-19T12:45:53Z

+
+TokenDecl: Modifier=TokenModifier? "token" Name=ID ":" Content=TokenElement Command=TokenCommand? ";"?
+
+interface TokenElement {}


WDYT about this name? Similar RegexpTokenElement -> RegexpTokenContent etc.

Suggested change

interface TokenElement {}

interface TokenContent {}

msujew · 2026-08-19T12:58:21Z

+}
+
+func (s *scopeProviderImpl) ScopeRuleCallRule(ctx context.Context, reference *core.Reference[AbstractRule]) core.Scope {
+	root, _ := reference.Owner().Document().Root.(Grammar)


Suggestion: Don't discard the ok of the cast. Can theoretically lead to panics here.

msujew · 2026-08-19T13:00:48Z

+func mergeImports(target *map[string]bool, source map[string]bool) {
+	for imp := range source {
+		(*target)[imp] = true
+	}
+}


Suggestion: Use maps.Copy for this. Also *map is not necessary. map is always a pointer type.

msujew · 2026-08-19T13:02:24Z

+								severity,
+								fmt.Sprintf("The token '%s' is never referenced in a parser rule, so the lexer can never produce it.", tokenRef.Name()),
+								member,
+								core.WithTextRange(member.TextRange()),


Suggestion: NewDiagnostic automatically takes the TextRange() value to initialize the range. No need to use WithTextRange again.

msujew · 2026-08-19T13:04:47Z

+			if !seen.Has(terminal.Name()) {
+				accept(core.NewDiagnostic(
+					severity,
+					fmt.Sprintf("The token '%s' is never referenced in a parser rule, so the lexer can never produce it.", terminal.Name()),


Can we move this string somewhere? It's repeated 6 times in this method.

msujew · 2026-08-19T13:05:57Z

+}
+
+func generateNestedString() (string, error) {
+	content := ""


Suggestion: Use a strings.Builder instead of a plain string here. Concatinating a new string 10000 times is quite expensive.

msujew reviewed Jul 14, 2026

View reviewed changes

Comment thread internal/generator/lexer_generator.go Outdated

Comment thread internal/generator/lexer_util_test.go Outdated

Comment thread internal/grammar/validator.go Outdated

Comment thread internal/languages/token_modes/parser_test.go

Lotes force-pushed the lotes/lexer-modes branch from 043764e to 0813295 Compare July 14, 2026 15:25

msujew mentioned this pull request Jul 27, 2026

Implement multi-language support #130

Open

msujew reviewed Jul 27, 2026

View reviewed changes

Comment thread internal/grammar/grammar.fb Outdated

Lotes force-pushed the lotes/lexer-modes branch from 467be1d to 2996cda Compare August 4, 2026 14:56

github-actions Bot reviewed Aug 4, 2026

View reviewed changes

Lotes marked this pull request as ready for review August 6, 2026 14:10

Lotes requested a review from msujew August 6, 2026 14:11

Lotes commented Aug 12, 2026

View reviewed changes

Comment thread token.go

Lotes commented Aug 12, 2026

View reviewed changes

msujew reviewed Aug 12, 2026

View reviewed changes

msujew and others added 15 commits August 13, 2026 13:19

WIP

372d5a0

Generate code

783ca2f

# Conflicts: # internal/grammar/types_gen.go # Conflicts: # internal/grammar/types_gen.go

Make my grammar changes

56544a1

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Rename Token

df2e77f

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Go back some steps

a6d990a

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Intermediate save

2d1b3f2

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io> # Conflicts: # lexer/lexer.go

Quick save

8519c9d

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Quick save

632e383

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Last corrections

02aa2e7

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add main generated files

3c6ec5b

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix examples

2711af2

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix token groups internal language

b04b0aa

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix lookahead internal language

c2e7ac8

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix completion internal language

29da360

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Quick save

d8b5018

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes added 26 commits August 13, 2026 13:25

Revert Fastbelt main grammar

e9cbc14

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Shorten completion grammar

4a65240

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add more unique rule name validations

7499f0d

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix rebase

1a03eba

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Rename GroupType to TokenModifier

40f2d9d

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix recovery and check token names for nil

efb5b0e

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add diagnostic for required default token mode

b1269aa

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add new validations

fb2bd56

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add more tests

10dcc56

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix token_modes.fb STRING_CONTENT rule

30ffbf3

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Persist more decisions

b6c72e9

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add code about non-cross-file-token-modes

4373189

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Update docs

a7fcf72

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove TODO.md

ccc62a4

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Revert statemachine and arithmetics example

a65e2fc

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Renamed groups and token types to token modifiers

75a8725

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove unrelevant capacity, remove parser from lexer benchmark

e3eb7fd

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Use WaitGroup.Go(...)

051248c

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Remove doc.FindAstNode(...)

be87ad5

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add test for unreachable token mode cycle

9dae3d7

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add more validations

a6383d4

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add validations

d0ecac4

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Fix tests

823998d

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Add another validation

3e0f309

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Removed the prefix from state number enum

71d3b2f

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Finish rebasing with MAIN branch

557bdc7

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes force-pushed the lotes/lexer-modes branch from 17d7c0e to 557bdc7 Compare August 13, 2026 11:37

Remove unused private function

032e9ff

Signed-off-by: Markus Rudolph <markus.rudolph@typefox.io>

Lotes mentioned this pull request Aug 13, 2026

Feature proposals on keywords #146

Open

msujew reviewed Aug 19, 2026

View reviewed changes


		TokenDecl: Modifier=TokenModifier? "token" Name=ID ":" Content=TokenElement Command=TokenCommand? ";"?

		interface TokenElement {}

Uh oh!

Uh oh!

Conversation

Lotes commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msujew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lotes commented Jul 17, 2026

Uh oh!

Lotes commented Jul 17, 2026

Uh oh!

msujew commented Jul 27, 2026

Uh oh!

Uh oh!

github-actions Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

Uh oh!

Lotes commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Lotes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msujew left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msujew left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Lotes commented Jul 10, 2026 •

edited

Loading

github-actions Bot left a comment •

edited

Loading

Lotes commented Aug 6, 2026 •

edited

Loading