Show automatically detected regressions and improvement

[alexcrichton]

Although the data can be noisy I've found it pretty obvious when there's a regression or an improvement. Basically any persisted regression for 3+ builds (or improvement) is typically here to stay. Would be awesome if there were a "trophy case" for improvements and a way to easily scan for regressions. Ideally we could tag regressions with upstream rust-lang/rust issues, but that's probably a bit too fancy!

[tgnottingham]

For a visual aid, see the persistent spike at the end of this graph of max-RSS:

Here's the perf results for the PR that caused the spike:

From the perf results, it looks like it might just be noise, since RSS stats are so noisy in general, but from the graph, the regression is very clear.

Edit: I realize now this isn't exactly the kind of case the OP is referring to. This is actually an example of a case where the regression isn't within noise for this particular benchmark. These still get missed routinely however, because the variance of a particular benchmark often isn't taken into account when looking at perf results. A 5% change in max-rss can be routine for one benchmark, and a clear regression for another. This case is easy to detect by looking at the recent historical variance for the particular benchmark and stat.

The case I think the OP is talking about is when you have a regression within noise, so it's not apparent it's a regression until you see that the graph has stabilized around a new high water mark. The graph is noisier in these cases, so we can't always easily identify the culprit.

Until we start automatically detect these regressions, we ought to at least scan the graphs for them when doing triage.