Notes on porting buf-list's tests from proptest to hegel

[sunshowers]

Hi! As promised on Hacker News I've ported over some property-based tests in my buf-list crate from proptest to hegel. Wanted to provide an experience report.

Background

The buf-list crate provides a segmented list of byte buffers. One of buf-list's features is a cursor type which provides io::Read etc on a BufList, just like std::io::Cursor does on contiguous sequences of bytes. The cursor type is very fiddly to get right, with binary searches and +/- 1 offsets scattered all over the place. Luckily it is extremely amenable to model-based testing because we have a very clear model: std::io::Cursor!

The PBT does:

• Create a segmented list of bytes with various sizes and contents, and wrap that in a buf_list::Cursor. This forms the SUT.
• Glue the bytes together into a contiguous sequence and wrap that in std::io::Cursor. This forms the oracle.
• Generate random sequences of operations. This is where the PBT framework (proptest/hegel) is really supposed to help.
• Run the operations by the SUT and the oracle; ensure behavioral equivalence (the results of each operation are the same) and that invariants aren't violated.

This is a pretty normal way to test this kind of interface. The PBT was also very high value at the time I was writing that code; it found no fewer than six bugs. Neither of these would be a surprise to y'all.

With proptest, the PBT looks like this. This is overall not the worst thing in the world, but the one issue is how proptest::sample::Index has to be used. Because the operations to be generated are data-dependent on the buf-list size, in order to avoid a flat-map one has to use Index fields everywhere. This kind of sucks, and I was hoping that Hegel's imperative model would do a much better job than that.

Porting the PBT

To port this test over to Hegel, my workflow was installing the Claude skill, then telling it:

/hegel port over the existing proptests to Hegel. Note the uses of proptest::sample::Index which can be replaced with imperative draws

Overall, it worked great! Took just a few minutes to do and the result was passing PBTs. This is the final result. I manually introduced bugs in a few spots and Hegel both caught them and provided a pretty good shrunk case.

One note was that Claude generated an integer and then mapped that to a variant. I changed it to writing out a discriminant enum with DefaultGenerator in sunshowers-code/buf-list@a655aa9. Because the values associated with the enum variants are data-dependent, I don't believe we can derive DefaultGenerator on CursorOp directly. In cases like mine, preferring a discriminant enum over an integer is probably preferable -- might be worth updating the Claude skill to tell it to do that.

Anyway, thanks again for all your work! This is really really exciting.

[Liam-DeVoe]

Thanks Rain, this kind of report is super helpful!

Because the values associated with the enum variants are data-dependent, I don't believe we can derive DefaultGenerator on CursorOp directly.

Because cursor_ops depends on a drawn num_bytes you mean? You're right this precludes using DefaultGenerator without modifications, but you can still #[derive(DefaultGenerator)] on CursorOp, and then define a configured generator inside of the test:

#[derive(DefaultGenerator)]
enum CursorOp {
    SetPosition(u64),
    ...
}

#[hegel::test]
fn cursors(tc: hegel::TestCase) {
    let num_bytes = tc.draw(...);
    let cursor_ops = default::<CursorOp>()
        .SetPosition(generators::integers::<usize>().max_value(num_bytes * 5 / 4) as u64)
        ...
    let cursor_op = tc.draw(cursor_ops);
}

This uses the derived generator for any fields you didn't configure. Of course, if you're going to configure every field, then the automatically-derived generators don't get used. There's still a benefit to using #[derive(DefaultGenerator)] on the enum, though, which is it exposes this inline syntax for configuring enum generators! This should remove the need for your CursorOpKind because you no longer need to track which enum field you chose to generate. That all gets handled declaratively.

I am only now realizing that this is an ergonomics win from #[derive(DefaultGenerator)] which isn't needed in e.g. Python, which doesn't allow you to attach data to enums. I wonder if we want a separate notion for "I want ergonomic configuration of enums but don't care about deriving default generators". (e: see #150)

A meta comment: I typically write this kind of model-based test as a stateful test, where each action is a #[rule] and you assert equivalence of the models in an #[invariant]: https://docs.rs/hegeltest/latest/hegel/stateful/index.html. Maybe this can't be written this way for whatever reason, or maybe proptest didn't have great stateful support? Hegel definitely intends for stateful testing to be a first class and well supported citizen, maybe even more so than Hypothesis. (We're not there yet though!)

I'd love if you continued to report any places where you feel hegel is worse than proptest, and of course any bugs or feature requests you find in hegel ❤️.

[sunshowers]

This uses the derived generator for any fields you didn't configure. Of course, if you're going to configure every field, then the automatically-derived generators don't get used. There's still a benefit to using #[derive(DefaultGenerator)] on the enum, though, which is it exposes this inline syntax for configuring enum generators! This should remove the need for your CursorOpKind because you no longer need to track which enum field you chose to generate. That all gets handled declaratively.

Ok this is awesome! I gave it a shot at sunshowers-code/buf-list#18.

The need for #![expect(non_snake_case)] is a bit surprising to me. I'd consider turning enum variant names into snake_case instead.

I am only now realizing that this is an ergonomics win from #[derive(DefaultGenerator)] which isn't needed in e.g. Python, which doesn't allow you to attach data to enums. I wonder if we want a separate notion for "I want ergonomic configuration of enums but don't care about deriving default generators". (e: see #150)

This would be perfect.

A meta comment: I typically write this kind of model-based test as a stateful test[...]

Thanks for the pointer! To be honest I've hesitated to use stateful PBT frameworks in the past because I've felt the magic to value ratio is a bit too high for me compared to standard PBT -- explicitly generating the list of operations to perform has generally been a smoother experience for me personally. It's quite possible Hegel is different enough though!

[Liam-DeVoe]

The need for #![expect(non_snake_case)] is a bit surprising to me. I'd consider turning enum variant names into snake_case instead.

Hmm, I agree the CamelCase method names are unfortunate, but auto-renaming could cause conflicts if an enum has eg both fields A and a. We could provide some sort of escape hatch in the macro for renaming the resulting methods, but I still don't love it

To be honest I've hesitated to use stateful PBT frameworks in the past because I've felt the magic to value ratio is a bit too high for me compared to standard PBT

Fair! :) You're missing out on stuff like swarm testing for stateful rules though!

[sunshowers]

I asked Claude to investigate the feasibility of converting this to hegel::state_machine and it pointed out three things:

1. The current macro implementation does not project #[cfg()] on #[rule] instances (buf-list uses it to guard a Tokio-specific enum variant/#[rule]) -- the macro needs to carry forward the same #[cfg()] annotations down to their implementations.
2. https://docs.rs/hegeltest/0.3.4/src/hegel/stateful.rs.html#216 sets a max step cap of 50. That's probably ok for us but maybe it should be configurable?
3. For state machines that borrow data, you can write impl Foo<'_> but impl<'a> Foo<'a> produces errors.

My current WIP is at sunshowers-code/buf-list#19. I do like the look a lot and I don't think issues 2 and 3 are blockers for me, but issue 1 definitely is.

[sunshowers]

Hmm, I agree the CamelCase method names are unfortunate, but auto-renaming could cause conflicts if an enum has eg both fields A and a. We could provide some sort of escape hatch in the macro for renaming the resulting methods, but I still don't love it

Definitely worth detecting this case and producing a good error message, but (speaking as someone who's written my fair share of proc macros) I think it will be extremely rare. Almost all enums in the real world use PascalCase for variant names. I'd just use the heck library or similar.

Fair! :) You're missing out on stuff like swarm testing for stateful rules though!

Interesting, this is the first argument I've heard that's made me reconsider. Is this something that's already implemented in Hegel?