Add CI tests for bracket

[maheshejs]

• started by adding turnt.toml used by brench to compare interpreter results (brili and exprs-lang's interpreter).
• next is configuring github actions similar to Add CI tests for MLIR dialect #465

[sampsyo]

Awesome. I was able to reproduce your instructions from the README and everything seems to work!

One not-very-important aside: at first, I was getting a crash (with the message "I/O operation on closed file") when running Brench on your config. I'm not sure why this only happened for me, but I tracked it down to a bug in Brench. I've fixed that in #468.

And a more-important suggestion: currently, your configuration file for Brench lives in a file called turnt.toml. This is a little odd, because you'd expect such a file to contain the configuration for Turnt (which is different, even though they're confusingly similar because they both reflect my weird tastes in testing tools).

It is a little nonstandard to use Brench as a testing tool, exactly; it was originally meant for benchmarking only. But of course it has to do a little bit of differential testing just to check for optimizations that break correctness. From that perspective, this is a perfectly reasonable "abuse" of Brench and I don't object to it on principle.

But anyway, with all that taken together, I think we should either:

1. rename turnt.toml to brench.toml and call it done
2. consider seeing what a Turnt-based testing setup would look like (probably by checking in .out files for each test)

I'm happy to go with either one!

[maheshejs]

i see, i just updated turnt.toml for turnt instead of brench.

[sampsyo]

Cool cool! I've tweaked this one step farther in the most recent commit, 38adb23. The idea is to avoid using two different output files (*.exprs_out and *.bril_out) for each program; we now just use one *.out file for both of them. This requires both pipelines to produce exactly the same output, which is what we want for a differential testing scenario.

Fortunately, everything still looks good:

$ turnt *.rkt
1..30
ok 1 - ackermann.rkt interp_exprs
ok 2 - ackermann.rkt interp_bril
ok 3 - ackermann.rkt bril
ok 4 - catalan.rkt interp_exprs
ok 5 - catalan.rkt interp_bril
ok 6 - catalan.rkt bril
ok 7 - delannoy.rkt interp_exprs
ok 8 - delannoy.rkt interp_bril
ok 9 - delannoy.rkt bril
ok 10 - fib_recursive.rkt interp_exprs
ok 11 - fib_recursive.rkt interp_bril
ok 12 - fib_recursive.rkt bril
ok 13 - fitsinside.rkt interp_exprs
ok 14 - fitsinside.rkt interp_bril
ok 15 - fitsinside.rkt bril
ok 16 - gcd.rkt interp_exprs
ok 17 - gcd.rkt interp_bril
ok 18 - gcd.rkt bril
ok 19 - loopfact.rkt interp_exprs
ok 20 - loopfact.rkt interp_bril
ok 21 - loopfact.rkt bril
ok 22 - mccarthy91.rkt interp_exprs
ok 23 - mccarthy91.rkt interp_bril
ok 24 - mccarthy91.rkt bril
ok 25 - recfact.rkt interp_exprs
ok 26 - recfact.rkt interp_bril
ok 27 - recfact.rkt bril
ok 28 - triangle.rkt interp_exprs
ok 29 - triangle.rkt interp_bril
ok 30 - triangle.rkt bril

Anyway, let me know if that all looks good to you, and we can merge this!

[maheshejs]

Anyway, let me know if that all looks good to you, and we can merge this!

Yes, looks good!

[sampsyo]

OK, cool! It is done!