Benchmark: Improved stats, now also printing stats for each individual test

[Michal-Mikolas]

For more detailed info about benchmark and for better benchmark debugging purpose I've added feature to print more detailed stats after the benchmark is done (or when running benchmark.py --stats).

The change is backward compatible with the current results, no need to re-run the old benchmarks to see these new details.

How does it look like

...

That's it. This change will give us better understanding of the model's strengths and weaknesses regarding different languages or info for investigating why one user got different score for the same model than other user.

[CLAassistant]

All committers have signed the CLA.

[Mushoz]

Really useful change! In my humble opinion, it would be awesome if it could also show stats on a per-language basis. In my experience, the programming ability can vary wildly from language to language. If I am looking for a strong model for coding in Python, then looking at the Go or Java tests is pretty useless.

[Michal-Mikolas]

Really useful change! In my humble opinion, it would be awesome if it could also show stats on a per-language basis. In my experience, the programming ability can vary wildly from language to language. If I am looking for a strong model for coding in Python, then looking at the Go or Java tests is pretty useless.

I didn't test it, but there is this parameter in the benchmark source code:

        "--stats-languages",
        help="Only include stats for specific languages (comma separated)",

[ziemkowski]

@Michal-Mikolas I think the ask, which would be neat to see, would be at the end of that list of individual tests, like this perhaps...

---- breakdown ---- pass/fail timeouts syn_err user_asks malformed exhausted error lazy ind_err
     cpp/...        6/14      0        0       21        2         0         3     0    0
     go/...         3/20      0        0       15        3         0         2     0    0
     ...
     rust/...       18/3      0        0       12        5         0         2     0    0

Maybe with pass % too?