Confusing reporting of undefined tokens

[The-Futurist]

I might have just resolved this, but I wanted to submit it anyway because it does look like the extension has a bug and/or could perhaps report the issue better than it does.

I found that when I added a literal token to the grammar file like { (that also happened to be already defined in the lexer g4) then reran antlr to generate the .tokens files then the extension behaves as described below.

I don't know whether 1. adding a literal token to a grammar file when we have a lexer file, is wrong or 2. adding a literal token when that same token is defined and named in a lexer g4, is the cause of the issue BUT antlr itself seems to accept this and will process the grammar and build the parser code!

So if the token g4 has this:

and we change the grammar file from this:

to this, saving the change

then rerunning antlr (to regen the .tokens files) then closing and opening the grammar g4 file, leads to the errrors:

I forgot that I'd even defined { as a token so that's my oversight BUT antlr does not complain, it accepts the two g4 files and builds the parser code so it's possible anltr has a bug...

Whatever it is I think the .tokens file(s) are what are upsetting the extension.

By the way, does the extension refresh when the .tokens files get regenerated? I work by editing the grammar/lexer, saving them (but not closing them) and then testing them by running antler and other test tools. So does the extension hand this? must I close and reopen the grammar/lexer files whenever I make a change to them and regen the .tokens files?

This the issue was about to post just now when I stumbled upon the cause:

=============================================================================

I'm coming back to some work after a couple years and I'm seeing VS Code report undefined tokens in my grammar.

Bot the grammar and lexer are in the same folder and I can run antlr and build the parser code, but in VS Code it reports some of the tokens as undefined and when I hover over them it seems to know where they are defined!

In addition the antlr test GUI tools also reports undefined tokens yet as I say, running antlr itself against the g4 files generates two .tokens files and all of the expected parser (and visitor etc) source code.

Also the problem seems to be intermittent, but far from clear, sometime I open the file and its fine but the tiniest edit sets of the undefined tokens problem. Also it only complains about some of the tokens in lexer file:

And as you can see here USES is flagged as undefined in the first image but further down the file it's no longer flagged as undefined!

I have version 2.4.7 of the extension installed.