Source generation support

[sebastienros]

@lahma @jbevain just to show you how it's done.
The logic is not part of the source generator and is driven by user code execution, not code syntax.
The source generator has to run the code first to get the object describing the grammar. Then evaluates the generated tree, not the code itself. That requires the code to be in a static method but that's totally fine. If the grammar needs options then one can generate different parser for each option, and reuse grammar portions across.

[sebastienros]

@gpetrou 1.6.0-preview.3 is being published with IncludeUsings attribute.

[sebastienros]

@Bykiev I am not sure your benchmark is even using the generated source. Current status:

    [GenerateParser]
    [IncludeUsings(
        "NCalc.Domain",
        "NCalc.Exceptions",
        "NCalc.Visitors",
        "System.Globalization",
        "System.Collections.Concurrent"
        )]
    [IncludeFiles(
        "src/NCalc.Core/Domain/BinaryExpression.cs",
        "src/NCalc.Core/Domain/BinaryExpressionType.cs",
        "src/NCalc.Core/Parser/Function.cs",
        "src/NCalc.Core/Domain/Identifier.cs",
        "src/NCalc.Core/Domain/LogicalExpression.cs",
        "src/NCalc.Core/Domain/LogicalExpressionList.cs",
        "src/NCalc.Core/Domain/TernaryExpression.cs",
        "src/NCalc.Core/Domain/UnaryExpression.cs",
        "src/NCalc.Core/Domain/UnaryExpressionType.cs",
        "src/NCalc.Core/Domain/ValueExpression.cs",
        "src/NCalc.Core/Domain/ValueType.cs",

        "src/NCalc.Core/Exceptions/NCalcParserException.cs",
        "src/NCalc.Core/Exceptions/NCalcException.cs",
        "src/NCalc.Core/Exceptions/NCalcFunctionNotFoundException.cs",
        "src/NCalc.Core/Exceptions/NCalcParameterNotDefinedException.cs",
        "src/NCalc.Core/Exceptions/NCalcParserException.cs",

        "src/NCalc.Core/Visitors/ILogicalExpressionVisitor.cs",

        "src/NCalc.Core/Parser/LogicalExpressionParserOptions.cs",
        "src/NCalc.Core/Parser/ArgumentSeparator.cs",
        "src/NCalc.Core/Parser/LogicalExpressionParserContext.cs"
        )]
    public static Parser<LogicalExpression> CreateExpressionParserComma()
    {
        return CreateExpressionParser(CultureInfo.InvariantCulture, ',');
    }

As you see I am trying to apply the generation root on the main grammar, so everything gets generated as part of it, instead of have most parsers be standard and invoke very minimal ones as generated (not much gain otherwise).

Still getting errors I am looking into. I may add back the mode where all the files of the current project are part of the compilation phase for the source generation. NCalc is quite complex (some options are questionable).

I can see the sergio + test failing too, could be an issue with the LeftAssociative parser.

[Bykiev]

@sebastienros, thank you for your inputs, indeed, in my PR most of the time GenerateParser attribute is used for standard parsers, because there was no IncludeUsings attribute. But the main restriction is the static method should be parameterless, which is not an easy task for NCalc as it using a lot of culture-specific stuff

[gpetrou]

I was able to use 1.6.0-preview.3 with IncludeUsings and IncludeFiles.
My remaining issue seems to be that there is still an error when referencing a type from an external DLL.
In that case, there is a warning PARLOT001: Emit failed for method 'Parser': The type or namespace name 'ExternalType' could not be found (are you missing a using directive or an assembly reference?)

[sebastienros]

@gpetrou slowly rewriting msbuild in Parlot ... Added support for glob patterns in IncludeFiles already after playing with Ncalc. I believe it's already including all referenced projects. Is the missing type from a PackageReference?

[gpetrou]

Is the missing type from a PackageReference?

Yes, it is.

[sebastienros]

@Genteure I fixed all the possible compilation issues, at least it works on NCalc which relies on compiled regexes, logging source gen and PolySharp. Just saying that Parlot's source generator will also invoke other source generator so it can build and run the host assembly so it can generate new code for this same host assembly. Not sure how to feel about it, but it seems to work. I am pushing a new version right now.

@Bykiev Made some progress and hitting a problem with the outer LeftAssociative parser that throws an exception, I don't understand what it does, neither does the source gen ;)

[Genteure]

I was watching the repository so this is already in my inbox, but appreciate the update 🙂
Perhaps you meant to ping @gpetrou

[gpetrou]

Should the new version work with System.Text.RegularExpressions.Generator?
There is a warning PARLOT012: Could not find generator assembly 'System.Text.RegularExpressions.Generator' specified in [IncludeGenerators]. Make sure the package is referenced in your project.
Since this is part of the framework, I assume that there is no need for a package.

[sebastienros]

@gpetrou ignore this one. Parlot stubs this one, and logger generator too. It should just work, my benchmark project had one too and it works.

[gpetrou]

Oh, I see. OK thanks.
I found one more issue.
If I have something like the following (just for demonstration purposes):

private abstract class NodeBase
{
}

private class BasicNode<T> : NodeBase
{
    public BasicNode(T value)
    {
        Value = value;
    }

    public T Value { get; }
}

private static readonly Parser<long> Long = Terms.Number<long>(NumberOptions.Integer);
private static readonly Parser<string> Equal = Terms.Text("==");

private static Parser<NodeBase> CreatePropertyParser<TComparand>(
   string name,
   Parser<string> @operator,
   Parser<TComparand> comparand)
{
    return
        comparand
            .AndSkip(@operator)
            .AndSkip(Terms.Text(name, caseInsensitive: true))
            .And(comparand)
            .Then<NodeBase>(
                    items => new BasicNode<TComparand>(
                        items.Item2));
}

[GenerateParser]
private static Parser<NodeBase> Parser()
{
    return CreatePropertyParser(
        "long",
        Equal,
        Long);
}

It seems to be missing the <TComparand> in the generated code.

[sebastienros]

@gpetrou I added a test with you sample, it's working fine.

NCalc is also working correctly without crazy changes. All the things that needed to be altered were described with the analyzer so it was smooth. The most copmlicated was to make the grammar change based on external options, since this can't be done when the parser is constructed anymore, but it all works out with the available parsers combinators (Select). ncalc/ncalc#533