Using grammar fragments

[hallvard]

I'm writing a textual syntax for a something similar to UML class diagrams. I have builtin/native types and classes that both start with a keyword, which is followed by an optional documentation string, zero or more tagged values and then the name, e.g.

BuiltinType: 'builtin' (documentation = STRING)? (tags += Tag)* name = ID

This parses both

builtin String as java java.lang.String

and

builtin "Native String" String as java java.lang.String

Since the documentation and tags are repeated in both rules, I tried to move them into a fragment rule, i.e.

fragment MetaData: (documentation = STRING)? (tags += Tag)*;

BuiltinType: 'builtin' MetaData name = ID

Note, this is a bit simplified, there are reasons the name part weren't also moved into the fragment.

But it didn't work, I got errors in my test code. I tried again and again, switching between a fragment and the exact same fragment code included directly. And the error was there when using the fragment, but not when including the fragment's text directly. How can this be, when the documentation says "[fragments] can be understood as being textually inserted where they are referenced".

My only guess is that there is some kind of indirection in the parser that doesn't work, since MetaData can consume nothing.

[msujew]

Hey @hallvard,

can you share a bit more of your grammar, especially the Tag rule? I tried reproducing it on our playground, but failed to do so, likely because the Tag rule is missing.

Based on your examples, I would assume that the Tag rule starts with an ID token. With that in mind, you're likely running into a parser issue that essentially all LL parsers have - which is that it's lacking outer context when determining whether or not to enter an optional element. We have an issue for that in the respective lookahead plugin repository.

[hallvard]

You're correct, a tag starts with an ID, and is then followed by either '!' or '=' (and a value) to distinguish from the name case.

But this shouldn't make a difference if fragments are 'textually' included where they are referenced? It should either work (or not) in both cases.

[msujew]

You're correct, a tag starts with an ID, and is then followed by either '!' or '=' (and a value) to distinguish from the name case.

In that case, you're very likely hitting the issue that I've mentioned.

But this shouldn't make a difference if fragments are 'textually' included where they are referenced? It should either work (or not) in both cases.

Internally, it still generates a parser rule for the fragment rule. It's just that the AST node is "preserved" across the two rules, which results in the combined fields from both rules on a single AST node. From the parser standpoint, whether something is a fragment rule or not doesn't matter.

[hallvard]

So the parser is slightly different, but flattening the AST makes it looks as if the rule was "textually included". Perhaps the documentation should have a note about the issue?