How to properly integrate ariadne into classic lexer/parser workflow?

[mainrs]

Hello!

This is a rather broad question. But I am having trouble figuring out how to do this properly. I have a lexer that turns my input into tokens that my parser consumes. The parser generates an AstNode tree.

Let's say I have the following input:

require "fileinto";                              # valid, since the argument is a string.
require ["imap4flags", "fileinto"];  # valid, since the argument is a list of strings.
require 5;                                           # invalid, since the argument has to be either a string or a string-list.

And my parser does the following:

fn require_cmd() -> impl Parser<Token, AstNode, Error = Simple<Token>> {
    let args = raw_string()
        .map(RequireArgs::Single)
        .or(raw_string_list().map(RequireArgs::Multiple));
    just(Token::Require)
        .ignore_then(args)
        .then_ignore(just(Token::Semicolon))
        .map(AstNode::RequireCommand)
        //.map_err_with_span(|_, span| {
        //    Simple::custom(span, "require accepts either a string or string-list")
        //})
}

fn raw_string() -> impl Parser<Token, String, Error = Simple<Token>> {
    select! { Token::Str(s) => s }.labelled("string")
}

fn raw_string_list() -> impl Parser<Token, Vec<String>, Error = Simple<Token>> {
    raw_string()
        .chain(just(Token::Comma).ignore_then(raw_string().repeated()))
        .or_not()
        .flatten()
        .delimited_by(
            just(Token::LeftSquareBracket),
            just(Token::RightSquareBracket),
        )
        .labelled("string-list")
}

pub enum AstNode {
    RequireCommand(RequireArgs),
}
pub enum RequireArgs {
    Single(String),
    Multiple(Vec<String>),
}

pub enum Token { Str(String) }

How would I go about generating some useful error message stating that the require command only accepts a string or a string-list? I think my problem is that I do not know how to properly "connect" your two libraries.

Edit: I went ahead and tried different things and ended up with the code that I commented out above. However, this gives me wrong span information. The span that I get for the message is 0..1 and not 8..9.

Edit 2: I think I just realized that the spans are in terms of tokens and not characters, aren't they? Is there an idiomatic way to map the token spans back to character spans?

[zesterer]

How would I go about generating some useful error message stating that the require command only accepts a string or a string-list?

For such cases, the usual approach is usually to have the parser accept a more permissive syntax than the language actually allows and then have a subsequent pass over the AST that does semantic checks (in most languages, that would be type-checking, making sure identifiers exist in the scope, etc. but for your language it would be checking arguments are valid, say).

Alternatively, you can perform a limited but sufficient form of this with the validate combinator.

I think I just realized that the spans are in terms of tokens and not characters, aren't they?

Spans are in terms of whatever you make them in terms of. If you're calling .parse(...) on an array-like type, then yes: chumsky will use array indices. However, if you use Stream::from_iter you can instead customise the spans of each tokens and chumsky will work out how to use these spans to synthesise new ones as appropriate.

[mainrs]

Spans are in terms of whatever you make them in terms of. If you're calling .parse(...) on an array-like type, then yes: chumsky will use array indices. However, if you use Stream::from_iter you can instead customise the spans of each tokens and chumsky will work out how to use these spans to synthesise new ones as appropriate.

That worked, thanks! For those wondering: it is important to pass your span to Stream::from_iter.
And thanks for your second hint. I will move my implementation to work on more generic input.

I do have one further question: does ariadne support rendering into JSON? Let's say if another program wants to consume my parser error outputs.

[zesterer]

It does not yet, no. There is an open issue for implementing something like this, but I'm not convinced that ariadne is even the 'correct' place to implement this: rather, I see ariadne as just one of many possible compiler diagnostic front-ends, with a JSON one being an alternative case.

[mainrs]

Thank you for your help again! I appreciate it!