End grammar rules using EOL?

[PatrikN-work]

Hi,

I want to write a grammar for the following type of structure,

define Commands
  commandName1 : command1 param param param
  commandName2 : command2
  commandName3 : command3 param 
enddef Commands

I want to terminate ever 'line' in 'Comamnds' by EOL but how.

I found somewhere that I can use:

terminal NEWLINE: /\\r?\\n/;
However, I cant get it to work.

I have this grammar (fragment) so far:

CommandsSection:
    'define' 'Commands'
    (CommandName ':' CommandList)+
    'enddef' 'Commands';

CommandName:
    comName=ID;

CommandList:
    (commands+=CommandDef)+ NEWLINE+;

CommandDef:
    cName=ID;

[msujew]

Hey @PatrikN-work,

I've adjusted your language to work as expected with regards to EOL markers. See this playground link. You can see what kind of AST the input produces by toggling the syntax tree viewer.

Note that I would generally recommend against employing newline-delimited languages, as they don't behave very well when it comes to error recovery behavior in Langium. Unless you're reimplementing an existing language in Langium, you're almost always better suited using a different delimiter like ;. Generic parsers are generally pretty bad at handling unexpected input in whitespace-aware languages. It also makes maintenance on the grammar pretty difficult, as you always need to keep track where a newline (or multiple newlines) is required/optional/not allowed.

[PatrikN-work]

Thanks,

Yes, i would not use newline as a delimiter but I'm implementing support for old existing languages/files so I can't really change it.

I was thinking of creating a "proper" grammar and generate the needed file but than I would need a serializer to add roundtrip support.

Any ETA on the serializer? :-)

[msujew]

Yes, i would not use newline as a delimiter but I'm implementing support for old existing languages/files so I can't really change it.

I see, that's unfortunate, but we've all been there :)

Any ETA on the serializer? :-)

None currently. No one of the contributing parties is interested enough in this, so there are no resources available to implement this.

I was thinking of creating a "proper" grammar and generate the needed file but than I would need a serializer to add roundtrip support.

Note that we usually just write a custom DSL serializer that serializes from the AST to a DSL in a similar way how you would write a DSL -> Java/C++/etc. transpiler. No need for framework support for this.