Skip to content

Commit f284302

Browse files
DonachDonach
committed
feat: add Grill JVM benchmark command
1 parent 7f5b838 commit f284302

10 files changed

Lines changed: 1821 additions & 61 deletions

File tree

‎README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,50 @@ The command exits with code `0` when dependencies are up to date and `1` when up
7070
> grill outdated
7171
```
7272

73+
### Benchmarking Wurst functions
74+
75+
`grill benchmark` measures Wurst code in the JVM-hosted Wurst IL interpreter. A benchmark is a package-level, parameterless function annotated with `@benchmark` and returning `int`; its return value must be a stable, workload-derived checksum.
76+
77+
```wurst
78+
import Wurstunit
79+
80+
@test @benchmark function benchmarkName() returns int
81+
var checksum = 0
82+
for i = 0 to 999
83+
checksum += i
84+
checksum.assertEquals(499500)
85+
return checksum
86+
```
87+
88+
The checksum should depend on the work being measured, not on a clock, random value, object identity, or mutable global state. When comparing two implementations, use the same inputs, operation count, and checksum calculation in both functions. A changing checksum indicates a correctness or benchmark-design problem and causes the comparison to fail.
89+
90+
For very small operations, batch many fixed inputs inside one benchmark invocation and return one checksum for the complete batch. This makes the measured workload large enough to distinguish implementations; the runner also reports the calibrated invocation `batchSize` for each fork. A benchmark may also carry @test. In normal test mode its assertions run and the int return is ignored; in benchmark mode the same return is the checksum. Use benchmark-only functions when running the workload during every test suite would be too expensive.
91+
92+
Run all benchmarks or select package/function names with an optional substring filter:
93+
94+
```cmd
95+
> grill benchmark
96+
> grill benchmark Polygon
97+
> grill benchmark Polygon --forks 5 --warmup 5 --iterations 20
98+
> grill benchmark Polygon --format json
99+
> grill benchmark --help
100+
```
101+
102+
Options are:
103+
104+
- `[filter]` — optional substring used to select benchmark names.
105+
- `--forks N` — positive number of isolated compiler JVMs per benchmark, serially (default `3`).
106+
- `--warmup N` — non-negative number of unmeasured warmup samples per fork (default `5`).
107+
- `--iterations N` — positive number of measured samples per fork (default `10`).
108+
- `--format human|json` — concise comparison output or machine-readable `wurst-benchmark-v1` JSON (default `human`).
109+
- `--help` — show benchmark-specific help without loading the project.
110+
111+
Global options such as `-projectDir`, `--quiet`, and `--debug` remain available.
112+
113+
Use JSON when a script needs raw samples, checksums, statistics, and environment metadata; diagnostics are kept off JSON stdout. The `environment.compiler` field is `sha256:<lowercase hex>` for the exact compiler JAR used by the workers. For credible relative results, keep the machine, OS, Java runtime, compiler and Grill versions, project inputs, fork/warmup/iteration settings, and background load consistent. Pin the process to dedicated CPU cores and avoid thermal or power-state changes where practical; `grill benchmark` does not itself control CPU affinity, frequency scaling, garbage collection, or other host-level noise.
114+
115+
Benchmark results measure the JVM-hosted Wurst IL interpreter. They vary with the machine, JVM, compiler version, host load, and benchmark setup. Use them for controlled side-by-side comparisons under the same conditions, not as absolute Warcraft III, Jass, Lua, or in-game performance numbers.
116+
73117

74118
### Building the project
75119

‎src/main/kotlin/benchmark/BenchmarkCoordinator.kt‎

Lines changed: 246 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
1+
package benchmark
2+
3+
import com.fasterxml.jackson.databind.JsonNode
4+
import com.fasterxml.jackson.databind.json.JsonMapper
5+
import java.nio.file.Files
6+
import java.nio.file.Path
7+
import java.util.Comparator
8+
import java.util.LinkedHashMap
9+
10+
class BenchmarkCoordinator(
11+
private val launcher: BenchmarkProcessLauncher,
12+
commonArguments: List<String>,
13+
private val debug: Boolean = false,
14+
private val compilerIdentity: String = "unknown",
15+
private val grillIdentity: String = "unknown"
16+
) {
17+
private val commonArguments = commonArguments.toList()
18+
private val mapper = JsonMapper.builder().build()
19+
20+
fun run(request: BenchmarkRequest): BenchmarkReport {
21+
val temporaryRoot = Files.createTempDirectory("grill-benchmark-")
22+
try {
23+
val discoveryOutput = temporaryRoot.resolve("discovery.json")
24+
val filterArguments = request.filter
25+
?.takeIf { it.isNotBlank() }
26+
?.let { arrayOf("-benchmarkFilter", it) }
27+
?: emptyArray()
28+
val discoveryArguments = workerArguments(
29+
"-runbenchmarks",
30+
"-benchmarkList",
31+
*filterArguments,
32+
"-benchmarkOutput",
33+
discoveryOutput.toString()
34+
)
35+
val selected = launchAndRead(discoveryArguments, discoveryOutput, "discovery", ::parseDiscovery).value
36+
37+
val forksByBenchmark = LinkedHashMap<String, MutableList<BenchmarkFork>>()
38+
val checksums = mutableMapOf<String, Int>()
39+
var executionIndex = 0
40+
repeat(request.forks) { forkRound ->
41+
val roundNames = if (forkRound % 2 == 0) selected else selected.asReversed()
42+
roundNames.forEach { qualifiedName ->
43+
val output = temporaryRoot.resolve("execution-$executionIndex-$forkRound-${safeFileName(qualifiedName)}.json")
44+
executionIndex++
45+
val arguments = workerArguments(
46+
"-runbenchmarks",
47+
"-benchmarkName",
48+
qualifiedName,
49+
"-benchmarkWarmup",
50+
request.warmup.toString(),
51+
"-benchmarkIterations",
52+
request.iterations.toString(),
53+
"-benchmarkOutput",
54+
output.toString()
55+
)
56+
val parsedExecution = launchAndRead(arguments, output, "execution for $qualifiedName") {
57+
parseExecution(it, qualifiedName, request.iterations)
58+
}
59+
val result = parsedExecution.value
60+
val previousChecksum = checksums.putIfAbsent(qualifiedName, result.checksum)
61+
if (previousChecksum != null && previousChecksum != result.checksum) {
62+
if (!parsedExecution.diagnosticsEmitted) {
63+
parsedExecution.diagnostics.forEach(System.err::println)
64+
}
65+
check(false) { "benchmark checksum changed across forks for $qualifiedName" }
66+
}
67+
val forks = forksByBenchmark.getOrPut(qualifiedName) { mutableListOf() }
68+
forks += BenchmarkFork(forkRound + 1, result.batchSize, result.samplesNanos)
69+
}
70+
}
71+
72+
val aggregates = selected.map { qualifiedName ->
73+
val forks = forksByBenchmark[qualifiedName].orEmpty().toList()
74+
check(forks.size == request.forks) {
75+
"benchmark $qualifiedName did not produce ${request.forks} fork results"
76+
}
77+
val allSamples = forks.flatMap { it.samplesNanos }
78+
BenchmarkAggregate(
79+
qualifiedName = qualifiedName,
80+
checksum = checksums.getValue(qualifiedName),
81+
forks = forks,
82+
statistics = BenchmarkStatistics.fromSamples(allSamples)
83+
)
84+
}
85+
if (aggregates.size == 2) {
86+
check(aggregates[0].checksum == aggregates[1].checksum) {
87+
"benchmark checksums differ: ${aggregates[0].qualifiedName}=${aggregates[0].checksum}, " +
88+
"${aggregates[1].qualifiedName}=${aggregates[1].checksum}"
89+
}
90+
}
91+
return BenchmarkReport(
92+
environment = BenchmarkEnvironment(
93+
os = System.getProperty("os.name", "unknown"),
94+
jvm = System.getProperty("java.version", "unknown"),
95+
compiler = compilerIdentity,
96+
grill = grillIdentity,
97+
cpuCount = Runtime.getRuntime().availableProcessors()
98+
),
99+
filter = request.filter,
100+
forks = request.forks,
101+
warmup = request.warmup,
102+
iterations = request.iterations,
103+
benchmarks = aggregates
104+
)
105+
} finally {
106+
deleteTree(temporaryRoot)
107+
}
108+
}
109+
110+
fun run(filter: String?, forks: Int, warmup: Int, iterations: Int): BenchmarkReport {
111+
return run(BenchmarkRequest(filter, forks, warmup, iterations))
112+
}
113+
114+
private fun workerArguments(vararg extra: String): List<String> {
115+
val arguments = commonArguments.toMutableList()
116+
if (!arguments.contains("-compactOutput")) {
117+
arguments += "-compactOutput"
118+
}
119+
arguments += extra
120+
return arguments
121+
}
122+
123+
private fun <T> launchAndRead(
124+
arguments: List<String>,
125+
output: Path,
126+
phase: String,
127+
parse: (JsonNode) -> T
128+
): ParsedWorker<T> {
129+
val result = launcher.run(arguments)
130+
var diagnosticsEmitted = false
131+
fun emitDiagnostics() {
132+
if (!diagnosticsEmitted) {
133+
result.output.forEach(System.err::println)
134+
diagnosticsEmitted = true
135+
}
136+
}
137+
if (debug) {
138+
emitDiagnostics()
139+
}
140+
if (result.exitCode != 0) {
141+
emitDiagnostics()
142+
}
143+
check(result.exitCode == 0) { "$phase worker exited with code ${result.exitCode}" }
144+
if (!Files.isRegularFile(output)) {
145+
emitDiagnostics()
146+
check(false) { "$phase worker did not write ${output.fileName}" }
147+
}
148+
val document = try {
149+
mapper.factory.createParser(Files.readString(output)).use { parser ->
150+
val parsed = mapper.readTree<JsonNode>(parser)
151+
check(parser.nextToken() == null) { "$phase worker wrote trailing JSON content" }
152+
parsed
153+
}
154+
} catch (exception: Exception) {
155+
emitDiagnostics()
156+
throw IllegalStateException("$phase worker wrote malformed JSON", exception)
157+
}
158+
if (document == null || !document.isObject) {
159+
emitDiagnostics()
160+
check(false) { "$phase worker JSON must be an object" }
161+
}
162+
return try {
163+
ParsedWorker(parse(document), result.output.toList(), diagnosticsEmitted)
164+
} catch (exception: Exception) {
165+
emitDiagnostics()
166+
throw exception
167+
}
168+
}
169+
170+
private fun parseDiscovery(document: JsonNode): List<String> {
171+
requireExactFields(document, setOf("schema", "mode", "benchmarks"), "discovery")
172+
check(document.requiredText("schema") == BENCHMARK_WORKER_SCHEMA) { "discovery worker schema is invalid" }
173+
check(document.requiredText("mode") == "discovery") { "discovery worker mode is invalid" }
174+
val benchmarks = document["benchmarks"]
175+
check(benchmarks.isArray) { "discovery worker benchmarks must be an array" }
176+
val names = benchmarks.map {
177+
check(it.isTextual && it.textValue().isNotBlank()) { "discovery benchmark name must be non-empty text" }
178+
it.textValue()
179+
}
180+
check(names.size == names.distinct().size) { "discovery returned duplicate benchmark names" }
181+
check(names.isNotEmpty()) { "benchmark discovery selected no benchmarks" }
182+
return names
183+
}
184+
185+
private fun parseExecution(document: JsonNode, expectedName: String, iterations: Int): WorkerExecution {
186+
requireExactFields(
187+
document,
188+
setOf("schema", "mode", "qualifiedName", "checksum", "batchSize", "samplesNanos"),
189+
"execution"
190+
)
191+
check(document.requiredText("schema") == BENCHMARK_WORKER_SCHEMA) { "execution worker schema is invalid" }
192+
check(document.requiredText("mode") == "execution") { "execution worker mode is invalid" }
193+
check(document.requiredText("qualifiedName") == expectedName) {
194+
"execution worker name does not match requested benchmark $expectedName"
195+
}
196+
val checksum = document["checksum"]
197+
check(checksum.isIntegralNumber && checksum.canConvertToInt()) { "execution checksum must be an integer" }
198+
val batchSize = document["batchSize"]
199+
check(batchSize.isIntegralNumber && batchSize.canConvertToInt() && batchSize.intValue() > 0) {
200+
"execution batchSize must be positive"
201+
}
202+
val samplesNode = document["samplesNanos"]
203+
check(samplesNode.isArray && samplesNode.size() == iterations) {
204+
"execution samplesNanos must contain exactly $iterations samples"
205+
}
206+
val samples = samplesNode.map {
207+
check(it.isIntegralNumber && it.canConvertToLong() && it.longValue() >= 0) {
208+
"execution samplesNanos must contain non-negative integers"
209+
}
210+
it.longValue()
211+
}
212+
return WorkerExecution(checksum.intValue(), batchSize.intValue(), samples)
213+
}
214+
215+
private fun requireExactFields(document: JsonNode, fields: Set<String>, label: String) {
216+
val actual = document.fieldNames().asSequence().toSet()
217+
check(actual == fields) { "$label worker JSON fields are invalid: expected $fields, got $actual" }
218+
}
219+
220+
private fun JsonNode.requiredText(field: String): String {
221+
val value = this[field]
222+
check(value != null && value.isTextual) { "worker field $field must be text" }
223+
return value.textValue()
224+
}
225+
226+
private fun safeFileName(name: String): String = name.replace(Regex("[^A-Za-z0-9_.-]"), "_")
227+
228+
private fun deleteTree(path: Path) {
229+
if (!Files.exists(path)) return
230+
Files.walk(path).use { stream ->
231+
stream.sorted(Comparator.reverseOrder()).forEach { Files.deleteIfExists(it) }
232+
}
233+
}
234+
235+
private data class WorkerExecution(
236+
val checksum: Int,
237+
val batchSize: Int,
238+
val samplesNanos: List<Long>
239+
)
240+
241+
private data class ParsedWorker<T>(
242+
val value: T,
243+
val diagnostics: List<String>,
244+
val diagnosticsEmitted: Boolean
245+
)
246+
}

‎src/main/kotlin/benchmark/BenchmarkModels.kt‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
package benchmark
2+
3+
const val BENCHMARK_WORKER_SCHEMA = "wurst-benchmark-worker-v2"
4+
const val BENCHMARK_SCHEMA = "wurst-benchmark-v1"
5+
const val BENCHMARK_DISCLAIMER =
6+
"Benchmark results measure the JVM-hosted Wurst IL interpreter. They vary with the machine, JVM, compiler version, host load, and benchmark setup. Use them for controlled side-by-side comparisons under the same conditions, not as absolute Warcraft III, Jass, Lua, or in-game performance numbers."
7+
8+
fun interface BenchmarkProcessLauncher {
9+
fun run(arguments: List<String>): BenchmarkProcessResult
10+
}
11+
12+
data class BenchmarkProcessResult(val exitCode: Int, val output: List<String>)
13+
14+
data class BenchmarkRequest(
15+
val filter: String? = null,
16+
val forks: Int = 3,
17+
val warmup: Int = 5,
18+
val iterations: Int = 10
19+
) {
20+
init {
21+
require(forks > 0) { "forks must be positive" }
22+
require(warmup >= 0) { "warmup must be non-negative" }
23+
require(iterations > 0) { "iterations must be positive" }
24+
}
25+
}
26+
27+
data class BenchmarkEnvironment(
28+
val os: String,
29+
val jvm: String,
30+
val compiler: String,
31+
val grill: String,
32+
val cpuCount: Int
33+
)
34+
35+
data class BenchmarkStatistics(
36+
val mean: Double,
37+
val standardDeviation: Double,
38+
val min: Long,
39+
val max: Long,
40+
val median: Long,
41+
val p90: Long,
42+
val p95: Long
43+
) {
44+
companion object {
45+
fun fromSamples(samples: List<Long>): BenchmarkStatistics {
46+
require(samples.isNotEmpty()) { "at least one benchmark sample is required" }
47+
require(samples.all { it >= 0 }) { "benchmark samples must be non-negative" }
48+
val sorted = samples.sorted()
49+
val mean = samples.average()
50+
val variance = samples
51+
.map { sample ->
52+
val delta = sample - mean
53+
delta * delta
54+
}
55+
.average()
56+
fun nearestRank(percentile: Double): Long {
57+
val rank = kotlin.math.ceil(percentile * sorted.size).toInt().coerceAtLeast(1)
58+
return sorted[rank - 1]
59+
}
60+
return BenchmarkStatistics(
61+
mean = mean,
62+
standardDeviation = kotlin.math.sqrt(variance),
63+
min = sorted.first(),
64+
max = sorted.last(),
65+
median = nearestRank(0.50),
66+
p90 = nearestRank(0.90),
67+
p95 = nearestRank(0.95)
68+
)
69+
}
70+
}
71+
}
72+
73+
data class BenchmarkFork(
74+
val fork: Int,
75+
val batchSize: Int,
76+
val samplesNanos: List<Long>
77+
)
78+
79+
data class BenchmarkAggregate(
80+
val qualifiedName: String,
81+
val checksum: Int,
82+
val forks: List<BenchmarkFork>,
83+
val statistics: BenchmarkStatistics
84+
)
85+
86+
data class BenchmarkReport(
87+
val schema: String = BENCHMARK_SCHEMA,
88+
val disclaimer: String = BENCHMARK_DISCLAIMER,
89+
val environment: BenchmarkEnvironment,
90+
val filter: String?,
91+
val forks: Int,
92+
val warmup: Int,
93+
val iterations: Int,
94+
val benchmarks: List<BenchmarkAggregate>
95+
)

0 commit comments

Comments
 (0)