How to prevent rewinding after a try_map failure?

[enklht]

Hi! Thanks for this excellent crate.

I have a question about error handling when using choice and try_map.

The Problem

I'm trying to parse integer literals. The parser should handle two cases:

1. A standard integer (e.g., 123), parsed as i32.
2. A long integer with an l suffix (e.g., 123l), parsed as i64.

I've implemented this as choice((int, long)).
When I provided an input that is too large to fit in i64, I expected a "number too large" error from the long parser.
However, I got an error from the int parser instead.

Here is a minimal example:

use chumsky::prelude::*;

#[derive(Debug)]
enum Token {
    Int(i32),
    Long(i64),
}

fn main() {
    let int = text::int::<&str, extra::Err<Rich<char>>>(10)
        .then_ignore(any().filter(char::is_ascii_alphanumeric).not().rewind())
        .try_map(|s: &str, span| {
            s.parse::<i32>()
                .map(Token::Int)
                .map_err(|e| Rich::custom(span, e))
        });

    let long = text::int(10)
        .then_ignore(one_of("lL"))
        .then_ignore(any().filter(char::is_ascii_alphanumeric).not().rewind())
        .try_map(|s: &str, span| {
            s.parse::<i64>()
                .map(Token::Long)
                .map_err(|e| Rich::custom(span, e))
        });

    let literal = choice((int, long));
    // let literal = int.or(long); // yielded the same result.

    println!("{:?}", literal.parse("10000000000000000000000l")); // doesn't fit in i64
}

This produces the following output:

ParseResult { output: None, errs: [found ''l'' at 23..24 expected digit, or something else] }

The desired error would be something like "number too large to fit in target type" from the long parser's try_map call.

Analysis and Questions

My understanding is that the choice combinator operates as follows:

1. It tries the int parser.
   This parser consumes the digits and fails when it encounters the l, noting that it consumed the digits.
2. It then tries the long parser.
   This parser successfully consumes the digits and the l.
   However, the try_map call fails during the conversion, and forgets that it consumed the digits + l.
3. choice sees that int consumed the digits while long consumed none, and it selects the error from int.

This leads to my questions:

1. Is this analysis of the rewinding behavior correct?
2. How can I keep the consumption state of long even when try_map fails, so that choice can choose an informative error message?

Thanks for your help!

---

p.s. A related question about error spans with the regex feature

As an alternative, I tried using the regex feature.

Here is the code:

let int = regex::<&str, extra::Err<Rich<char>>>(r"[0-9]+\b").try_map(|s: &str, span| {
    s.parse::<i32>()
        .map(Token::ConstInt)
        .map_err(|e| Rich::custom(dbg!(span), e))
});

let long = regex::<&str, extra::Err<Rich<char>>>(r"[0-9]+")
    .then_ignore(regex(r"[lL]\b"))
    .try_map(|s: &str, span| {
        s.parse::<i64>()
            .map(Token::ConstLong)
            .map_err(|e| dbg!(Rich::custom(span, e)))
    });

let literal = choice((int, long));

println!("{:?}", literal.parse("10000000000000000000000l"));

This correctly produces the "number too large" error, but the span was 0..0:

[src/main.rs:21:30] Rich::custom(span, e) = number too large to fit in target type at 0..24
ParseResult { output: None, errs: [number too large to fit in target type at 0..0] }

The dbg! output shows the correct span (0..24) when the error is created.
Why does the final error reported by parse have a collapsed span of 0..0?

[zesterer]

What you're looking for is the validate combinator! It's similar to try_map, except it lets you emit non-fatal errors in addition to an output value.

Thanks for the note about the regex combinator. I believe this is an oversight, yes. I'll look into it.

[enklht]

Thank you for your suggestion!
Actually, I've considered the validate combinator, but in my scenario, I don't think that works since the conversion error should be fatal and immediately fail the current branch of the choice.
What I want to prevent is the rewinding of the span (or the consumption state?) after an error.

I will try to explain the issue briefly.
Suppose we have a parser parser.try_map(fallible_function).
If the fallible_function fails, it seems to report no input was consumed, even when the parser part has already consumed significant input.
This seems to prevent the choice combinator from choosing the right error to report, at least in my case.

[zesterer]

I think I see what you mean.

This seems to arise due to where in the input try_map attributes the error to. Right now, it attributes it to the start of the parsed input, which causes its error to be superseded by the error from the int parser, which occurs later on in the input (chumsky generally assumes that errors occurring further into the input are more useful for the end user).

There is... an inherent tension here. For some uses of try_map, attributing to the start is indeed the intended behaviour and produces the best results, so I don't think this is obviously a bug.

Out of interest, was the then_ignore(any().filter(char::is_ascii_alphanumeric).not().rewind()) added for any particular reason? I think that this might be something that's throwing off the error prioritisation here.

[enklht]

I see. I understand that this behavior is intentional, and I didn’t mean to suggest it was a bug.
However, I also think it would be useful to have some control over the consumption state of the parsers.

Regarding the then_ignore(any().filter(char::is_ascii_alphanumeric).not().rewind()), it’s there to perform a word boundary check. This makes each case mutually exclusive and prevents the parser from attempting conversions that are clearly destined to fail. In this case, the conversion is cheap, so omitting it would be fine, but for aesthetic reasons and to satisfy a specific test suite, I kept the boundary check.

---

To control the consumption state, I think a sequential parser combinator could be a good approach, something like combine(parsers: tuple-like).
This would differ from a .join chain or group in that it updates the consumption state only after all parsers have succeeded.

For example:
Suppose a and b parse "a" and "b" respectively. If you apply them sequentially to the input "ac",
a.join(b) and group((a, b)) would report that it consumed "a", while combine((a, b)) would report that it consumed nothing.

[zesterer]

I don't think there is a specific notion of 'consumption state' that exists as you might imagine it.

After doing some thinking, I think the correct way to solve it would be to associate an error not just with a location in the input, but a location span, which would allow us to correctly disentangle the try_map case more cleanly and hopefully satisfy both constraints. Unfortunately, this would likely have performance ramifications and would require some non-trivial changes to the guts of the crate. I'll open a long-term issue for the problem.

In the meantime, a relatively clean solution might be something like the following pseudocode:

text::int().then(just('l').or_not()).try_map(|(int, l)| if l.is_some() {
    // Try parsing into an i64
} else {
    // Try parsing into a i32
})

Let me know if that behaves as you expect!

[enklht]

Thank you!
Error handling with spans sounds promising, though it’ll need some careful discussion before introducing it. I’m looking forward to seeing how the crate develops.

Also, your solution worked quite well! (It got a little bit messy since I had to handle unsigned and float parsing too.)
Thanks again!