In our benchmark suite, @gdalle and I had several tests named similarly to named-tuples:
N = 2
K = 3
SUITE["Foo"]["(N=$N, K=$K)"] = @benchmarkable foo($x)
In the table generated by PkgTemplate's export_markdown, benchmarks like this get renamed to the ID ["Foo", (:(N = 2), :(K = 3))].
My best guess is that the string "(N=$N, K=$K)" somehow gets parsed as a named tuple.
The real problem is that all benchmarks with named-tuple-like names get excluded from the comparison table, regardless of improvements or regressions.
We ran into this issue here, where none of our ["bench_jogger.jl", "Hessian", ...] benchmarks appeared in the comparison. However, they did appear in the baseline and target sections with noticeable improvements, as shown in these screenshots. Note that all of them had named-tuple-like names.
In our benchmark suite, @gdalle and I had several tests named similarly to named-tuples:
In the table generated by PkgTemplate's
export_markdown, benchmarks like this get renamed to the ID["Foo", (:(N = 2), :(K = 3))].My best guess is that the string
"(N=$N, K=$K)"somehow gets parsed as a named tuple.The real problem is that all benchmarks with named-tuple-like names get excluded from the comparison table, regardless of improvements or regressions.
We ran into this issue here, where none of our
["bench_jogger.jl", "Hessian", ...]benchmarks appeared in the comparison. However, they did appear in the baseline and target sections with noticeable improvements, as shown in these screenshots. Note that all of them had named-tuple-like names.