Description
The performance regression test (as shown in https://www.mmtk.io/ci-perf-result/index.html) has several problems.
- It's using DaCapo 2006 for both OpenJDK and JikesRVM. This is bad because after 20 years people program differently. Its data set is also not large enough for modern machines.
- Heap sizes are too large. GC will not be triggered frequent enough to detect even major performance differences during GC, such as this one which was supposed to reduce the total execution time of SemiSpace by 10% and reduce GC time by 20%.
For OpenJDK, we should switch to DaCapo Chopin. We have to continue using DaCapo 2006 for JikesRVM because it only supports Java 6.
We should adjust the heap size to a reasonable value. For DaCapo Chopin, it should be a multiple of min heap size w.r.t. G1 in OpenJDK for each benchmark. It is also better to run several different heap sizes with fewer invocations for each size (such as 1.5x, 2x, 2.5x and 3x min heap size, 10 invocations each) instead of running one single heap size for too many times (such as 2.0x min heap size for 40 invocations).
And we also need to fix reference processing for OpenJDK. Currently, some benchmarks in DaCapo Chopin exhibits a strange behavior that the min heap size grows proportionally to the number of iterations. It can be something like 120MB, 180MB, 240MB, 300MB and 360MB for 1-5 iterations. That indicates some kind of memory leak, and is likely caused by the lack of reference processing. See: mmtk/mmtk-openjdk#304