[syntax] Support for metadata over iteration and dispatch statements.

[modulovalue]

There's no way to attach metadata to statements that have multiple valid compilation outputs. Consider:

forStatement
    :    AWAIT? FOR '(' forLoopParts ')' statement
    ;

whileStatement
    :    WHILE '(' expression ')' statement
    ;

doStatement
    :    DO statement WHILE '(' expression ')' ';'
    ;

switchStatement
    :    SWITCH '(' expression ')'
         LBRACE switchStatementCase* switchStatementDefault? RBRACE
    ;

This makes it rather difficult to run VM experiments that depend on different code generation patterns. For example, repetition statements such as the forStatement, doStatement or whileStatement could be annotated to be made eligible to (or forced to) unroll which is common in bit arrays, e.g. see FastBitSet.js

That is, the following is not possible today:

void fn() {
  @pragma("...unroll", 8)
  for (...) {}
}

Same for switchStatement, there are many valid generation patterns all with different trade-offs (e.g., computed gotos, branch tables, binary search, linear search). Some of that has been explored in dart-lang/sdk#49585. However, this optimization fails silently if the pattern diverges from what the compiler expects and performance becomes unpredictable. There's no way to easily lock in the optimization due to lack of support for metadata there.

Same here, the following is not possible today:

void fn() {
  @pragma("...computed-gotos")
  switch (...) {}
}

Proposal:

Support metadata on forStatement, whileStatement, doStatement, switchStatement.

forStatement
    :    metadata AWAIT? FOR '(' forLoopParts ')' statement
    ;

whileStatement
    :    metadata WHILE '(' expression ')' statement
    ;

doStatement
    :    metadata DO statement WHILE '(' expression ')' ';'
    ;

switchStatement
    :    metadata SWITCH '(' expression ')'
         LBRACE switchStatementCase* switchStatementDefault? RBRACE
    ;

[lrhn]

It can be meaningful to annotate, well, anything, and it's not a big technical issue. Most compilers just ignore annotations that are not for themselves.

Annotations are more user-friendly than comments. A comment like /*@pragma('vm:unroll')*/ could also work, if you can find it in the AST model, but you lose type checking and documentation. Or potential documentation if you use a less general annotation than pragma.

There are grammar issues that need to be handled.
The language ran into the first ones with declaration annotations when it introduced records, because then ( could start a declaration. statements and expressions can start with ( and < too.

Dart, and most C-derived languages, have a very dense expression syntax, where the same symbol can mean different things
in different contexts, depending on whether you start an expression or continue one. Examples:

• [e1] is a literal, e1[e2] is an index operator.
• -e1 is a negation, e1-e2 is subtraction.
• (e1) is grouping, e1(e2) is function invocation.

A valid expression syntax is also a continuation of another expression, and then it means something different.

Because of that, the grammar tries to ensure is that you can never have two adjacent expressions without a delimiter between them, because then it doesn't know whether to start or continue an expression.
Annotations end with an undelimited expression. Statements can start with an undelimited expression. We have a problem.

Examples:

• @A(B);. Is this @A(B) on the empty expression statement ; (we really should remove that from the language, just write {}!), or @A on the expression statement (B).
• @A(B)[C]; Is this @A(B) on the list literal [C] or @A on the index-operator invocation (B)[C], as expression statements?
• @A(B)(C);. Is this @A(B) annotating the expression statement (C); or @A annotating (B)(C).
• @A(B)-(C);. Is this @A(B) annotating the expression statement -(C); or @A annotating (B)-(C).
• @A<B>(C){D};: Is this @A<B>(C) on the block statement {D} (followed by an empty statement) or @A on the generic function literal <B>(C){D} as an expression statement? (We know a function literal as expression statement is useless, so we can choose to disambiguate, but it's still a parsing complication.)

SO if we are to annotate statements (or even expressions), I think we need to delimit the annotation or at least separate it from the following expression or statement.

Either put delimiters around the entire annotation (or even allow multiple annotations in the same braces:

• @(A(foo)) statement; - just grouping.
• @{A(foo), B()} statement; - A set of annotations, so you need only one.
• @[A(foo), B()] statement; - Or an ordered list of annotations.
• ``@A(foo) statement;` - ... no,

Also considered (@A(foo)), where (@ is the annotation started, and it's ended by a matching ), and everything inside
must be an annotation. Could work. It thing I'd end up writing (@A(foo) expr) far too often, but maybe it's easy enough to get used to. I don't think [@A(foo)] or {@A(foo}} would feel the same.

Both collections could allow general constant expressions, so:

@{if (bool.fromEnvironment('dart.library.io')) @pragma('vm:platform_const')}
final isLinux = Platform.current.isLinux;

We could allow this on declarations too.

Or, since the annotation already has a start character, just add an end delimiter;

• @A(foo): statement; (Like a label!)
• @A(foo); statement; (Like a prior statement. I very often write ; after my declaration annotations, so it must feel right. Except semicolon looks more like a separator, and the annotation shouldn't be separated from the thing it belongs to.)
• @A(foo)@ (That's awful, so no.)

I lean towards @{...} or @[...], with (@...) as a close third. Any of them will work, the @[...] suggests order matters, the @{...} does not
Only the annotation interpreter knows whether order matters, but it might, so @[...] is probably best. But it's also slightly more dense visually.

Let's see how they look:

void fn() {
  @(pragma("vm:computed-gotos"))
  switch (...) {}
}

void fn() {
  @{pragma("vm:computed-gotos")}
  switch (...) {}
}

void fn() {
  @[pragma("vm:computed-gotos")]
  switch (...) {}
}

void fn() {
  (@pragma("vm:computed-gotos"))
  switch (...) {}
}

void fn() {
  @pragma("vm:computed-gotos"):
  switch (...) {}
}

void fn() {
  @pragma("vm:computed-gotos");
  switch (...) {}
}

Visually, I like @{...}, @...: and (@..,.) best, because they're the least intrusive. The @(...) looks more like a function call than a grouping operation ... because standard Dart (and all similar language) formatting has spaces before ( as grouping, not space before ( as invocation, so @( looks like an invocation. We use whitespace to parse code visually, it really matters.

The (@...) is growing on me!

Also, if the annotation is completely delimited, nothing limits it from being any constant expression.
You could do (@bool.fromEnvironment('experiment') ? UseExperiment() : UseDefault()) ....
(That's not a new power, you could always do

const _annotation = @bool.fromEnvironment('experiment') ? UseExperiment() : UseDefault();
@_annotation 
... someDecl ...

It was only disallowed for parsing reasons.)

The delimiter will be required for all annotations on statements, even if @foo switch(...) doesn't need one for parsing.

It can work on individual expressions too, although associativity isn't clear, (@foo) a + b, is that (@foo)(a + b) or ((@foo) a) + b? Probably the latter, only allow it on a primary expression, but that has to be specified.

Or maybe expressions need to be fully parenthesized, so ((@foo) expr). Then (@foo: expr) starts looking better, so maybe we do want a separate syntax for expressions. We also don't have labeled expressions (yet!), so there is not the same precedence for where to put things that apply to it.

In that case, maybe @...: is a good label-like annotation for a statement, and (@...: expr) is a good well-delimited annotation for an expression. That would be more consistent. Don't think @{metadatum} expr generalizes that well.

Last thought: Would you ever want to annotate an annotation. (I did say "anything"!).
Then @...: needs to be delimited (it's an expression), so it'd be:

• @(@recordOccurrence:markerAnnotation): statement,

That's not actually annotating the annotation, it's annotating the annotation expression. Maybe that'll be good enough.
But if you can have any expression after the :, you might be tempted to do:

const _ = 0;
@(_: arbitrary constant expression here)

Might as well allow @(any constant expression).

Let's go for it! Which would mean a grammar like:

<metadatum> ::= 
       <qualified> (('.' <identifier>)? <typeArguments>? <arguments>)? 
     | <parenthesizedExpression>

<metadatumLabel> ::= '@' <metadatum> ':'

<parenthesizedExpression> ::= '(' <metadatumLabel>* <expression> ')'

<statement> ::= <metadatumLabel>* <label>* <unlabeledStatement>

<primary> ::= <parenthesizedExpression> | ... rest except "'(' <expression> ')'" ...

<element> ::= <metadatumLabel> <element> | ... existing ...

Can allow mixing statement metadata and labels.
If we ever add labels to expressions, it would probably go into parenthesized expression too,
and then we can just combine metadatum-label and label.

Tried adding to elements too, so [if (test) @all: ...list else @one: a + b].
If we change the grammar of if/while to use <parenthesizedExpression>, then you can write if (@random: someTest) ...
without needing extra parentheses.
Could also allow metadata directly on other already delimited expressions, like arguments, record values and interpolation expressions. (@foo: a, @bar: b), and not ((@foo: a), (@foo: b)). Probably worth it, similar to elements.

(Maybe aim lower, but I like to be prepared for everything, so we don't paint ourselves into a syntactic corner.)

[modulovalue]

Amazing analysis @lrhn, thank you!

Turning the label syntax into a general purpose statement annotation syntax seems like a very natural step for the language and I strictly prefer that over adding metadata in front of statements with no trailing delimiter.

Some statements unfortunately already support the existing metadata annotations:

localVariableDeclaration
    :    metadata initializedVariableDeclaration ';'
    |    metadata patternVariableDeclaration ';'
    ;

localFunctionDeclaration
    :    metadata functionSignature functionBody
    ;

So I suppose we would want to deprecate that for the new syntax.

but I like to be prepared for everything, so we don't paint ourselves into a syntactic corner.)

Note that some languages define what's essentially outer and inner metadata annotations. An outer metadata annotation attaches to the syntax tree parent, an inner metadata annotation to the immediate child. That's a dimension worth considering if we wanted a more "complete" metadata mechanism.

A different perspective of the inner/outer idea that applies to expressions is time: do we annotate the expression before it was evaluated or after? For that we might want pre/post metadata annotations:

// Pre-metadata annotated expression
const _ = @(Foo: 1 + 2 + 3) // Awesome!

// Post-metadata annotated expression
const _ = @(1 + 2 + 3 :Foo) // I don't like this as much.

Pre/post metadated expressions would provide unambiguous support for communicating pre/post conditions between code generators and code ingestion tools which is difficult to do without proper syntax.

[Wdestroier]

Could whitespace (' ', '\n', etc) be required? Similar to how whitespace is required in classUser {}.

1. Have a whitespace after the annotation name and its arguments if any.
2. Not having a whitespace between the annotation name and its arguments

// Correct:
@A
class B{}
// or
@A() class B {}

// Wrong:
@Aclass B{} // Error: missing new-line after @A.
// or
@A()class B{}

[lrhn]

Using whitespace is also an option, but it's not great. It's already done for declaration annotations.
I'd like to not make more whitespace signficant, and with new syntax, we may be able to do something better.
I don't want to make newlines significant, so it shouldn't matter which whitespace is used.

It's not about "correct" or "wrong", it's about which correct thing it means.
(And being predictable.)

So:

@A (B); // whitespace after `A`, so annotation is `@A` and statement is `(B);`.
@A(B); // No whitespace between `A` and `(`, so annotation is `A(B)` and statement is `;`.
@A <B>(C){}; // Annotation is `A` and statement is `<B>(C){};` (a useless generic function literal expression statement).
@A<B>(C){}; // Annotation is `A<B>(C)` and statement is `{}` (followed by `;`).
@A<B> (C){}; // Space between `>` and `(` is just not allowed, `@A<B>` is not a valid annotation, and the space means that's where it ends.