Need some help writing the parser

[stevefan1999-personal]

I know it's kinda shameless to ask but I can't really figure out why this couldn't parse.

You can inspect the program here: https://replit.com/@stevefan1999_pe/TinyLanguage. Do cargo test instead.

I had a feeling there is an indirect left recursion in statement, that's why it failed to parse {} and {;}

[kevinmehall]

The issue is extremely subtle: The lexer names the enum variant SemiColon but in the grammar you use [Semicolon]. The PEG [...] compiles into a Rust match expr like match t { Semicolon => ... }, which has the pitfall where unknown names are interpreted as a fresh variable that matches any value. Therefore the e:expr()? [Semicolon] in rule expression_statement() skips the optional expr() and matches the } with the [Semicolon] because of this. Then you're out of tokens and fail to match the [BracketCurlyRight] in block_statement where it's supposed to be.

Try cargo test parser::tests::test_block_statement --features peg/trace if you haven't seen the tracing feature.

We set allow(non_snake_case) in the generated code, which prevents Rust's normal warning about this. I'll need to look into if that allow can be narrowed to not prevent that warning. warning: variable Semicolon should have a snake case name is still pretty subtle, though...

[kevinmehall]

By the way, I don't see any actual left recursion here and I think the #[cache_left_rec] is unnecessary -- let_statement, if_statement, while_statement, block_statement all begin with a literal token so are all trivially not left recursive, and the base rules in expr()'s precedence!() also begin with specific tokens rather than recursing. (@ doesn't count as left recursion because it's implemented by precedence climbing so isn't actually recursive)

[stevefan1999-personal]

@kevinmehall Great. This is a really big unsight...
So is it also possible to use the captured value? Right now I have to do this:

        rule identifier() -> Ustr = [Identifier(iden)] { iden } / expected!("identifier")
        rule type_integer() -> Ustr = [TypeInteger(size)] { size } / expected!("<integer type>")
        rule type_string() -> Ustr = [TypeString] { "string".into() } / expected!("<string type>")
        rule type_bool() -> Ustr = [TypeBool] { "bool".into() } / expected!("<bool type>")
        rule type_char() -> Ustr = [TypeChar] { "char".into() } / expected!("<char type>")
        rule type_number() -> Ustr = [TypeNumber] { "number".into() } / expected!("<number type>")
        rule r#type() -> Ustr = identifier() / type_integer() / type_string() / type_bool() / type_char() / type_number()

Rather than this:

        rule r#type() -> Ustr
            = ([Identifier(iden)] / expected!("identifier")) { iden }
            / ([TypeInteger(size)] / expected!("identifier")) { size }
            / ([TypeString] / expected!("string")) { "string".into() }
            / ([TypeBool] / expected!("bool")) { "bool".into() }
            / ([TypeChar] / expected!("char")) { "char".into() }
            / ([TypeNumber] / expected!("number")) { "number".into() }

Or maybe let tokens to be self-descriptable, also though the use of another proc macro (that's why I love Rust so much). Right now I don't see any remap function for &[T] so you basically have to guess what the pattern match is, which is not hard for devs but hard for users

[kevinmehall]

The captured value is only in scope in the same side of / that captured it. (Otherwise what would the value be?) So you want

        rule r#type() -> Ustr
            = ([Identifier(iden)] { iden } / expected!("identifier"))
            / ([TypeInteger(size)] { size } / expected!("identifier"))
            / ([TypeString] { "string".into() } / expected!("string"))
            / ([TypeBool] { "bool".into() } / expected!("bool"))
            / ([TypeChar] { "char".into() } / expected!("char"))
            / ([TypeNumber] { "number".into() } / expected!("number"))

You should generally be able to take the definition of a rule, wrap it in parentheses and substitute it in place of the call to the rule in most cases. The exception is when the rule return type is needed for type inference.

Or maybe let tokens to be self-descriptable,

Not sure what you mean by this, but if you have functions or methods for your token type, you can call them from the grammar:

        rule r#type() -> Ustr
            = [t if is_type(t)] { t.to_string() }