Add Benchmark Hide-State to Perf-Plot URL

[VanDulti]

When having a large number of related benchmarks (in my case ~200), the perf plot does take quite a long time to load. This comes with Bencher Console becoming somewhat unresponsive.
Obviously, ~200 benchmarks are to many for a single plot at the same time, but Bencher supports hiding and unhiding plots - making it reasonable to hide all of them initially and offer the option to unhide relevant ones or even all of them when needed.
I would propose persisting the hide-state in a project's plots by making it accessible from the plot url. This would allow me to further customize plots in iframes too.

[epompeii]

@VanDulti thank you for using Bencher, and I'm sorry things aren't working as you expected.

The Perf Plots can support up to 255 permutations. However, as you've experienced this can take a while to load. I've done quite a bit of performance tuning on this front. Though I should probably take another look at it.

With that said, preventing this sort of usage was the primary motivation behind creating Pinned Plots. Have you tried setting up Pinned Plots instead of using one massive Perf Plot?

If so, what problems have you been running into and what makes you prefer the massive Perf Plot?

If not, could you help me understand why?

This would allow me to further customize plots in iframes too.

I'm curious, where are you using these iframes?

Is there a reason iframes for each set of the plot selections doesn't work?

[VanDulti]

Hi, thanks for the thorough answer & sorry to keep you waiting.

Yes, I definitely use pinned plots and they work well, but for this specific usecase, we have many related benchmarks that belong together. Plus, we do have unrelated measures/benchmarks that already take up pinned plots. It wouldn't make sense to create ~200 pinned plots imo, as that would completely shadow those other plots or make them hard to find.

In my opinion, the benchmark selection (hide/unhide plots) works quite well for my usecase. The only problem is the drop in performance and the obvious overwhelmingly large amount of lines in a single plot. By providing an option to hide them by default (and of course lazy-load the data/load it in the background), those issues could be fixed.

I'm using the iframes to be able to embed plots at specific timestamps/branches/... into another site.

Is there a reason iframes for each set of the plot selections doesn't work?

Recreating the benchmark-selection in my site and updating the iframe accordingly would of course be an option, but I am not a fan of duplicating logic just to make things work. Especially if you already have almost-perfect plot selection built into to perf plots ;)

[epompeii]

It wouldn't make sense to create ~200 pinned plots imo, as that would completely shadow those other plots or make them hard to find.

Would being able to group Pinned Plots change this at all?

See: #405

Note, there is currently a limit of 64 Pinned Plots per Project, so you would run into that limit as well.

I'm using the iframes to be able to embed plots at specific timestamps/branches/... into another site.

Can you help me understand, what is this other site?
The more detail the better. I'm trying to make sure I really understand your use case.

[VanDulti]

It wouldn't make sense to create ~200 pinned plots imo, as that would completely shadow those other plots or make them hard to find.

Would being able to group Pinned Plots change this at all?

See: #405

Being able to group pinned plots sounds like a useful thing, though I'm not sure it would help my specific usecase. It would definitely help against shadowing other plots. + it gives users a way to structure their dashboard view.

I'm using the iframes to be able to embed plots at specific timestamps/branches/... into another site.

Can you help me understand, what is this other site? The more detail the better. I'm trying to make sure I really understand your use case.

It's a kind of report-dashboard that 1. integrates bencher report info (perf plots including alerts) and 2. includes data about our product for being able to backtrack benchmarking-results to the problem (e.g. lots of logs). This means it's correlating benchmarking-results/reports from bencher with data that could have lead to the regressions, assigned to whoever is in charge of the "update/product".
For the perf plots I'm currently using iframes as provided by Bencher's share plot ui. What I'm doing is kind of similar to #562, altough for me to use that, I would at least need the option to fully configure report dashboards to include the plots that I want. For this, I would once again need the option to hide all benchmarks in that large plot by default and lazy load data when needed (for performance and usability). ~200 automatically generated plots would lead to the same problems as before, at least if that many benchmarks actually trigger alerts. Edit: in case of alerting a plot/alert could actually make sense

[epompeii]

Thank you for all of the details!

2. includes data about our product for being able to backtrack benchmarking-results to the problem (e.g. lots of logs).

Where are these logs generated? Where are they stored?
Is this dashboard just a link to the logs or does it present them to the user?

This means it's correlating benchmarking-results/reports from bencher with data that could have lead to the regressions, assigned to whoever is in charge of the "update/product".

Does your tool also handle the assigning and tracking of tasks to the responsible party or is that handled by another tool (Jira, GitHub Issues, etc)?

[VanDulti]

Thank you for all of the details!

2. includes data about our product for being able to backtrack benchmarking-results to the problem (e.g. lots of logs).

Where are these logs generated? Where are they stored? Is this dashboard just a link to the logs or does it present them to the user?

The logs are generated on the "testbed", i.e. by our software that gets benchmarked running on the testbed/our product. For starting out, we want to provide a bunch of data including the logs as a link to download, but the exact process of what is to be included in the site is yet to be evaluated. I do think that the exact way of backtracking failed benchmarks to the inherent problem is extremely dependent of the product/software that's developed -> Not sure if that should be in the scope of bencher as a general-purpose benchmarking tool.

This means it's correlating benchmarking-results/reports from bencher with data that could have lead to the regressions, assigned to whoever is in charge of the "update/product".

Does your tool also handle the assigning and tracking of tasks to the responsible party or is that handled by another tool (Jira, GitHub Issues, etc)?

Nope, not in scope for now - at least not automatically.

[epompeii]

I do think that the exact way of backtracking failed benchmarks to the inherent problem is extremely dependent of the product/software that's developed -> Not sure if that should be in the scope of bencher as a general-purpose benchmarking tool.

My current thinking is that Bencher should help with some of this. For your use case, I'm imagining you being able to add a link to the logs in the bencher run command: #127

So Bencher would remain agnostic to how those logs are generated and stored. It would just provide a simple mechanism for you to provide the key/value pair.