Introduce Comparison Column to Output Tables

[MontyCarter]

I'm not sure if this idea fits within the direction you guys would like to take BenchExec, however, I like to get my ideas written down because I forget them 15 minutes later if I don't. If this doesn't fit within the direction of BenchExec, please feel free to close this straight away.

Anyway, it might be useful to be able to compare two executions. What I have envisioned is an additional column for a table with two tool runs. In this additional column, the rows for each benchmark could detail information such as difference in runtime (or speedup as a ratio), memory usage, and discrepancy in execution result (correct vs wrong, etc). In the execution summary at the bottom, different metrics could be used for comparing the two executions as a whole, such as difference in aggregate runtime, aggregate speedup, aggregate accuracy, etc.

In fact, instead of building special tables with this additional column, two columns could be selected for comparison dynamically (possibly similar to a "Compare Items" functionality on an e-commerce website), and the execution comparison column can be displayed with javascript alone, without having to alter the output table from tablegenerator, since all required information is already present in the table. (That is, add the javascript code to compare execution columns to all tables generated by tablegenerator, and then users can activate the functionality by selecting two columns to compare.)

This idea has no urgency - I just thought it would be a nice enhancement.

[PhilippWendler]

This sounds interesting, and doing this dynamically using JavaScript would certainly be a easy way to integrate it. I am not sure whether I have the time to work on this in the near future, but I would certainly like to have this added.

For now, do you know all the features that provide small parts of what you propose?
For comparing column "status" (with the tool result), you can use the *.diff.html table that is generated if there is at least one row with differing results.
If you want to use this information programmatically (e.g., for regression testing), there is the parameter --dump to table-generator.
For comparing two numerical columns, you can use the scatter plots.
For comparing the aggregate results more easily than looking at the bottom of the table,
you can click on the cells in the left-most column of the table footer (e.g., the cell "total tasks"),
and you get an overview (this view should be exposed better probably).

[PhilippWendler]

For completeness, there is now another feature that provides a little part of this feature request: arbitrary columns can be used as the base for computing the "diff" tables with the relevantForDiff attribute of <column> tags.