Update parser as per latest changes in nom

[subhojit777]

No description provided.

[Hywan]

Thanks for the PR!

Do you have an idea why tests are failing?

[subhojit777]

I will take a look. I did cargo build, and I came to know to know that InputTake and InputTakeAtPosition are not implemented. I will look into the tests.

[subhojit777]

@Hywan So I found one of the many unknown reasons why the tests are failing.

One problem is that the new Span is not passing the correct offset, line, column values - after the split. I am looking for examples where, after the slice is converted into utf8, the values are computed using a function or something. There are tabs and newslines in the input, it should return the correct offset and line after converting it into utf8. Do you know any such example?

For instance:

    fn case_first_with_whitespace() {
        named!(hello<Span, Span>, tag!(b"hello"));
        named!(test1<Span, Span>, first!(tag!(b"hello")));
        named!(test2<Span, Span>, first!(hello));

        let input  = Span::new(b"  \nhello\t\r");
        let output = Ok((Span::new_at(b"\t\r", 8, 2, 6), Span::new_at(b"hello", 3, 2, 1)));

        assert_eq!(test1(input), output);
        assert_eq!(test2(input), output);
    }

Fails with out output:

Diff < left / right > :                                                                                                                                                     
 Ok(                                                                                                                                                                        
     (                                                                                                                                                                      
         Span {                                                                                                                                                             
<            offset: 0,                                                                                                                                                     
<            line: 1,                                                                                                                                                       
<            column: 1,                                                                                                                                                     
>            offset: 8,                                                                                                                                                     
>            line: 2,                                                                                                                                                       
>            column: 6,                                                                                                                                                     
             slice: [                                                                                                                                                       
                 9,                                                                                                                                                         
                 13                                                                                                                                                         
             ]                                                                                                                                                              
         },                                                                                                                                                                 
         Span {                                                                                                                                                             
<            offset: 0,                                                                                                                                                     
<            line: 1,                                                                                                                                                       
>            offset: 3,                                                                                                                                                     
>            line: 2,                                                                                                                                                       
             column: 1,                                                                                                                                                     
             slice: [                                                                                                                                                       
                 104,                                                                                                                                                       
                 101,                                                                                                                                                       
                 108,                                                                                                                                                       
                 108,                                                                                                                                                       
                 111                                                                                                                                                        
             ]                                                                                                                                                              
         }                                                                                                                                                                  
     )                                                                                                                                                                      
 )

[Hywan]

When computing the new Span, you should update the offset to include i (from find). Same for line and column.

You might want to check

parser/source/tokens.rs

Lines 1180 to 1222 in 077aa7a

	fn slice(&self, range: $range) -> Self {
	let next_slice = &self.slice[range];
	if next_slice == self.slice {
	return *self;
	}
	let next_offset = self.slice.offset(next_slice);
	if next_offset == 0 {
	return Span {
	offset: self.offset,
	line : self.line,
	column: self.column,
	slice : next_slice
	};
	}
	let consumed = &self.slice[..next_offset];
	let number_of_newlines = bytecount::count(consumed, b'\n') as u32;
	let next_column =
	if number_of_newlines == 0 {
	self.column + next_offset as u32
	} else {
	match memchr::memrchr(b'\n', consumed) {
	Some(last_newline_position) => {
	(next_offset - last_newline_position) as u32
	},
	None => {
	unreachable!();
	}
	}
	};
	Span {
	offset: self.offset + next_offset,
	line : self.line + number_of_newlines,
	column: next_column,
	slice : next_slice
	}
	}

to see an example of that logic. One way to do that would be (I imagine):

Ok((self.slice(i..), self.slice(..i))

Because slice will return a Span with appropriated values set.

[subhojit777]

I looked into the remaining failing tests. One of them is case_exclude_empty_set(). I found that it is failing because it uses the InputTakeAtPosition and FindToken implementations of &[u8] which is provided by the nom framework.

The test output:

Diff < left / right > :
 
<Err(
<    Incomplete(
<        Size(
<            1
<        )
>Ok(
>    (
>        [],
>        [
>            102,
>            101,
>            100,
>            97,
>            98,
>            99
>        ]
     )
 )

Do you think that they need to be implemented differently inside the parser library? @Hywan

[subhojit777]

    fn case_exclude_empty_set() {
        named!(
            test,
            exclude!(
                is_a!("abcdef"),
                alt!(
                    tag!("abc")
                  | tag!("ace")
                )
            )
        );

        assert_eq!(test(&b"fedabc"[..]), Ok((&b""[..], &b"fedabc"[..])));
    }

is_a!("abcdef") uses find_token() which returns bool. But split_at_position1() does Some() pattern match-ing. This is weird. I think I am missing something here. Any idea?

[Hywan]

Please, give me some days to look at this. I don't time right now, hope to get free time in few days :-). Thanks for your patience!

[subhojit777]

Meanwhile I can do more investigation.

#132 (comment) - I am wrong in this comment. It is alright there. The find() inside split_at_position1() expects the Fn to return a bool

But I am confused here because in every iteration it is true, still it is executing the None arm.

[Hywan]

Working on this right now.

[subhojit777]

@Hywan Did you check this?

[Hywan]

Yes, and I have similar issue. I've paused my patch for some weeks because of other projects. I'm planning to switch back to Tagua VM very soon.