Performance tests: clarify current usage and improve local usability

[vogella]

I would like to improve the performance tests, but first I want to confirm how they are used today.

My understanding:

• The performance tests are not part of our releng process and do not run in CI.
• They appear to exist only to be executed manually.

When I tried to run them manually for evaluating performance, I found them hard to use:

• The output is not easy to read or interpret.
• There is no clear distinction between setup time and actual test execution time.

Question before I start any work:

Is it correct that the performance tests are currently not used in our releng process and are only present for manual execution?
If not, how are they actually used?

cc @HannesWell and @HeikoKlare

[HeikoKlare]

To the best of my knowledge, performance tests are currently not executed in any automation. And I don't think anyone actively executes them manually. I just did for those tests I migrated to JUnit 5 to see if they still work in a somehow expected way. So in general, those tests are probably not used at all right now.

Since we at Vector are currently dealing with some significant performance issues, we have also been thinking about reviving and extending the performance tests for Eclipse to better find regressions in important use cases.
My vision would be that we execute those tests freqeuently (maybe once a week) on the dedicated machines we also use for I-Builds. This could, for example, be done on weekends when usually there is no load from PRs, I-Build tests and the like. Currently, we might have to limit the execution to those platforms we have dedicated agents for (Windows, MacOS) as the containerized execution for Linux may lead to instable performance results, but that's something we could check then.

I am not sure how the performance test results could be evaluated automatically. In our internal performance tests framework we have defined reference values for the dedicated performance test machines that we evaluate against (and then also have a database with Grafana dashboards to visualize the development). Maybe there is/was already a concept for the Eclipse performance tests, but I am not aware of that.

[vogella]

Thanks @HeikoKlare for the confirmation that the current state of performance tests is not usuable for you as well.

These days I typically have performance tests for before and after which I run locally before and after a commit. Output is something like this:

┌──────────────────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Theme swap │ min │ median │ p95 │ max │ mean │
├──────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Before (master) │ 2076 ms │ 2227 ms │ 2368 ms │ 2368 ms │ 2235 ms │
├──────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ After (Rework) │ 1841 ms │ 2011 ms │ 2116 ms │ 2116 ms │ 2009 ms │
├──────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Delta │ −235 │ −216 │ −252 │ −252 │ −226 │
├──────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ % │ −11% │ −9.7% │ −11% │ −11% │ −10% │
└──────────────────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┘

So I do not get information over time but at least I get information about a change. Currently I cannot use the Eclipse Performance tests for that purpose and it would be nice to have them usable for this scenario for me.

I start looking at the structure to see if I can improve them, for example I noticed that we still have references to the old IBM Derby database (dead since > 10 years) in our code, time to remove that I would say.

[HannesWell]

Is it correct that the performance tests are currently not used in our releng process and are only present for manual execution?

I can also confirm the first part of the question. Performance tests are currently not executed automatically anywhere in the RelEng Jenkins instance or anywhere else.
Since I've never ran them, I currently also don't know how they are set up and work. Therefore I cannot help with the second part(s). Sorry.

My current plan is to significantly streamline/simplfy/rework the current set up of the regular I/Y-build.
Potentially this will also affect/improve performance tests respectivly I can also consider them, if there is now a generally increased interested for them.

Currently, we might have to limit the execution to those platforms we have dedicated agents for (Windows, MacOS) as the containerized execution for Linux may lead to instable performance results, but that's something we could check then.

I second that.

I am not sure how the performance test results could be evaluated automatically. In our internal performance tests framework we have defined reference values for the dedicated performance test machines that we evaluate against (and then also have a database with Grafana dashboards to visualize the development). Maybe there is/was already a concept for the Eclipse performance tests, but I am not aware of that.

Besides setting hard limits (should be tight to capture even small regressions), also the trends could be evaluated if the execution is not 100% stable. Of course then one will only receive a failure delayed.

[laeubi]

You can run the performance tests at least for SWT by adding the performance label to a PR https://github.com/eclipse-platform/eclipse.platform.swt/blob/acfd8318c01db1d08d2b8985147080e0d200936b/.github/workflows/maven.yml#L96

I think they where actually removed because literally no one looked at the results. To get comparable measures one would likely need to run the test once with an old release and then compare them with the current I-Build, then variations per machine are not that important (but might still apply).

[vogella]

I think they where actually removed because literally no one looked at the results.

IIRC they were removed because, during development of the Eclipse 4 API, IBM reduced manpower and the DB backing the comparison eventually died, with no effort invested into developing an alternative. The times were also highly dependent on machine usage, so people ignored them anyway.

So AFAICS the performance tests are left in a state designed to be aggregated into a DB (which doesn't happen) rather than for human eyes, hence they typically provide no helpful information to the human tester.

When I need a heavy test scenario, I usually end up modifying the test output to issue useful timing information and typically add a @RepeatedTest(10) to get a reliable average including a warm-up phase.

[akurtakov]

The DB requirement has been removed years ago .
Last working state was having 2 runs - 1 with the baseline, 2 with the latest I-build and compare the times. Last this worked in 4.32 https://archive.eclipse.org/eclipse/downloads/drops4/R-4.32-202406010610/performance/performance.php but due to usage of VMs results were useless.

[HannesWell]

FYI the agent in the RelEng Jenkins instance dedicated for performance tests is now effectively shutdown respectively will probably not be revived as per https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/work_items/7600#note_7607066.
If performance tests are enabled, they have to be moved to another (static) agent, but I assume that this isn't a problem.

[akurtakov]

@MohananRahul Did I correctly tagged you in https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/work_items/7600#note_7606593 ?