Gradle skips the jmh task when attempting to run the benchmarks again

[daniel-rusu]

Describe the bug
When I attempt to run the jmh task a second time without making any code changes, Gradle skips running it. That's because Gradle sees that all the dependencies of the jmh task are unchanged so Gradle expects the output of the jmh task to be the same as the previous result and skips it.

To Reproduce

1. Update some code that the benchmarks use.
2. Run ./gradlew jmh. This will run the benchmarks successfully.
3. Attempt to run ./gradlew jmh again without making any changes
4. Bug: Gradle deems the jmh task as complete and skips it without attempting to run the benchmarks again

My Solution:
I'm not sure how to fix the gradle plugin but in case it helps, I fixed this problem on my end by adding the following to build.gradle.kts:

// Consider the output of the JMH task always out of date to allow running benchmarks again even if no code changed
tasks.withType<me.champeau.jmh.JMHTask> {
    outputs.upToDateWhen { false }
}

[melix]

I wouldn't say that this is a bug, but the expected behavior. Should you want to rerun the benchmark, you can use the --rerun flag.

[daniel-rusu]

@melix , unfortunately --rerun also reruns the task's dependencies so it's not a good solution as it rebuilds the entire project again and re-generates the benchmarks.

From a technical Gradle perspective, the task is misconfigured making Gradle cache the results believing that the output won't change when its dependencies haven't changed. This Gradle assumption is invalid as benchmarks have variability from run to run based on unexpected system load etc so the task should be performed whenever it's requested.

Fixing this also simplifies documentation so that users can be provided a single command for running the benchmarks as it would be misleading to specify --rerrun when running it for the first time and also wasteful and time-consuming to unnecessarily re-run task dependencies that have already completed (eg. build).

[melix]

@melix , unfortunately --rerun also reruns the task's dependencies so it's not a good solution as it rebuilds the entire project again and re-generates the benchmarks.

That is extremely suspicious. Are you confusing with --rerun-tasks? There's no reason --rerun would re-execute tasks dependencies, especially if re-running the benchmark is up-to-date without it. Either the task dependencies change, and then the task would be re-executed in any case, or, the task dependencies didn't change and only the benchmark would be re-executed.

[daniel-rusu]

Oops, sorry, I did confuse --rerun with --rerun-tasks as that's what I used before my "fix".

My other points still apply about cleaner documentation and the task being misconfigured according to Gradle standards. This configuration also encourages Gradle to cache the results of this task which it shouldn't do for benchmark results.