Skip to content

Commit 4eb3080

Browse files
IzaakwltnIzaakwltn
committed
Added parameterized benchmarks, convergence detection
1 parent e4303de commit 4eb3080

File tree

6 files changed

+450
-143
lines changed

6 files changed

+450
-143
lines changed

‎benchmarking/README.md

+80-24
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Coalton benchmarking prints to the repl by default.
1818
This setting can be turned off with:
1919

2020
```
21-
(cl:setf *coalton-verbose-benchmarking* cl:nil)
21+
(cell:write! *coalton-verbose-benchmarking* False)
2222
```
2323

2424
### Printing width
@@ -27,14 +27,14 @@ Coalton benchmarks print to the repl at 90 characters wide by default.
2727
This can be changed using:
2828

2929
```
30-
(cl:setf *coalton-benchmark-width* 90)
30+
(cell:write! *benchmark-width* <UFix>)
3131
```
3232

3333
### Print time in cientific notation
3434
By default, times are printed using scientific notation. This can be turned off using:
3535

3636
```
37-
(cl:setf *coalton-benchmark-sci-notation* cl:nil)
37+
(ell:write! *coalton-benchmark-sci-notation* False)
3838
```
3939

4040
## Defining benchmarks:
@@ -43,7 +43,7 @@ Benchmarks can be defined in any Coalton package (that imports or nicknames `#:c
4343

4444
```
4545
;; Defining a Coalton benchmark
46-
(define-benchmark stak 1000 ; iterations
46+
(define-benchmark stak 1000 ; samples
4747
(fn ()
4848
(stak 18 12 6)
4949
Unit))
@@ -56,27 +56,58 @@ Benchmarks can be defined in any Coalton package (that imports or nicknames `#:c
5656
Unit)))
5757
```
5858

59+
Parameterized benchmarks allow for multiple inputs to the same function:
60+
61+
```
62+
(define-parameterized-benchmark rec-fib 1000
63+
(fn (x)
64+
(fib x)
65+
Unit)
66+
(seq:make 10 15 20 25))
67+
68+
```
69+
Using `:detect-convergence?` will add standard deviation information to your run, as well as dynamic sample scaling.
70+
71+
### Dynamic sample scaling
72+
73+
74+
With `:detect-convergence?` selected, benchmarks will run one sample at a time, collecting live standard deviation information for timings:
75+
76+
```
77+
(define-parameterized-benchmark rec-fib-generic 1000
78+
(fn (x)
79+
(fib-generic-wrapped x)
80+
Unit)
81+
(seq:make 10 15 20 25)
82+
:detect-convergence? cl:t)
83+
84+
```
85+
86+
When running this mode, the `samples` input is interpreted as the minimum samples, and the benchmark will continue running until the standard deviation of time converges. This ensures the minimum amount of samples before a reliable average.
87+
88+
This mode is most effective on benchmarks with significant individual sample times.
89+
5990
## Running individual benchmarks
6091

6192
Individual benchmarks can be run with `#'run-benchmark`, as long as the benchmark is defined in the current package.
6293

6394
`#'run-benchmark` returns a `BenchmarkResults` object.
6495

6596
```
66-
COALTON-BENCHMARKS> (coalton (run-benchmark "tak"))
97+
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmark "rec-fib0"))
6798
┌─────────────────────────────────────────────────────────────────────────────────────────┐
68-
│ Benchmark tak │
99+
│ Benchmark rec-fib0 │
69100
├─────────────────────────────────────────────────────────────────────────────────────────┤
70101
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
71102
├─────────────────────────────────────────────────────────────────────────────────────────┤
72103
│ Coalton development mode without heuristic inlining │
73-
├───────────────────────┬─────────────────────┬─────────────────────┬─────────────────────┤
74-
│ Benchmark │ Time Elapsed │ Bytes consed │ # Iterations │
75-
├───────────────────────┼─────────────────────┼─────────────────────┼─────────────────────┤
76-
│ TAK │ 9.4079e-1 s │ 0 │ 1000 │
77-
└───────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘
104+
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
105+
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
106+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
107+
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
108+
└──────────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┘
78109
79-
#.(BENCHMARKRESULTS "TAK" 1000 940788 #.(SOME 0))
110+
#.(BENCHMARKRESULTS REC-FIB0 1000 2371 2 #.NONE #.(SOME 0))
80111
```
81112

82113
## Running package benchmarks
@@ -85,24 +116,49 @@ Package benchmarks can be run with `#'run-package-benchmarks`.
85116

86117
`#'run-package-benchmarks` returns a `PackageBenchmarkResults` object.
87118

119+
Local packages can be run with `#'run-benchmarks`.
120+
88121
```
89-
COALTON-BENCHMARKS> (coalton (run-package-benchmarks "coalton-benchmarks/gabriel/tak"))
122+
COALTON-BENCHMARKS/FIBONACCI> (coalton (run-benchmarks))
90123
┌─────────────────────────────────────────────────────────────────────────────────────────┐
91-
│ Package 'coalton-benchmarks/gabriel/tak' │
124+
│ Package 'COALTON-BENCHMARKS/FIBONACCI' │
92125
├─────────────────────────────────────────────────────────────────────────────────────────┤
93126
│ System: ARM64 OS-MACOSX SBCL2.2.4-WIP │
94127
├─────────────────────────────────────────────────────────────────────────────────────────┤
95128
│ Coalton development mode without heuristic inlining │
96-
├─────────────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┤
97-
│ Benchmark │ Run time │ Real time │ Bytes consed │ # Iterations │
98-
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤
99-
│ TAK │ 1.043406 s │ 1.044723 s │ 65520 │ 1000 │
100-
├─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤
101-
│ LISP-TAK │ 0.082777 s │ 0.082867 s │ 65520 │ 1000 │
102-
└─────────────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┘
103-
104-
#.(PACKAGEBENCHMARKRESULTS "coalton-benchmarks/gabriel/tak" #.(COALTON-BENCHMARKING/BENCHMARKING::BENCHMARKSYSTEM "ARM64" "OS-MACOSX" "SBCL" "2.2.4-WIP" COMMON-LISP:NIL COMMON-LISP:NIL) #(#.(BENCHMARKRESULTS "TAK" 1000 1040557 1041583 95888)
105-
#.(BENCHMARKRESULTS "LISP-TAK" 1000 83104 83040 65520)))
129+
├──────────────┬──────────────┬──────────────┬──────────────┬──────────────┬──────────────┤
130+
│ Benchmark │ Time (ms) │Avg Time (ms) │ Time std dev │ Space (B) │ # Samples │
131+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
132+
│ REC-FIB0 │ 2 │ 0 │ n/a │ 0 │ 1000 │
133+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
134+
│ REC-FIB1 │ 20 │ 0 │ n/a │ 0 │ 1000 │
135+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
136+
│ REC-FIB2 │ 216 │ 0 │ n/a │ 0 │ 1000 │
137+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
138+
│ REC-FIB3 │ 2370 │ 2 │ n/a │ 0 │ 1000 │
139+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
140+
│ REC-FIB-GENE │ 17 │ 0 │ 0.0055017565 │ 0 │ 1002 │
141+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
142+
│ REC-FIB-GENE │ 95 │ 0 │ 0.0053594956 │ 0 │ 1002 │
143+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
144+
│ REC-FIB-GENE │ 912 │ 1 │ 0.0226823009 │ 0 │ 1002 │
145+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
146+
│ REC-FIB-GENE │ 10270 │ 10 │ 0.3188679543 │ 0 │ 1014 │
147+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
148+
│ REC-FIB-LISP │ 105 │ 0 │ n/a │ 0 │ 1000 │
149+
├──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┤
150+
│ REC-FIB-MONO │ 306 │ 0 │ n/a │ 0 │ 1000 │
151+
└───────────────────────┴─────────────────────┴─────────────────────┴─────────────────────┘
152+
#.(PACKAGEBENCHMARKRESULTS "COALTON-BENCHMARKS/FIBONACCI" #.(BENCHMARKSYSTEM "ARM64" "OS-MACOSX" "SBCL" "2.2.4-WIP" COMMON-LISP:NIL COMMON-LISP:NIL) #(#.(BENCHMARKRESULTS REC-FIB0 1000 2476 2 #.NONE #.(SOME 0))
153+
#.(BENCHMARKRESULTS REC-FIB1 1000 20065 20 #.NONE #.(SOME 0))
154+
#.(BENCHMARKRESULTS REC-FIB2 1000 215897 216 #.NONE #.(SOME 0))
155+
#.(BENCHMARKRESULTS REC-FIB3 1000 2370219 2370 #.NONE #.(SOME 0))
156+
#.(BENCHMARKRESULTS REC-FIB-GENERIC0 1002 16914 17 #.(SOME 5.501756508182156d0) #.(SOME 0))
157+
#.(BENCHMARKRESULTS REC-FIB-GENERIC1 1002 94676 94 #.(SOME 5.359495667149465d0) #.(SOME 0))
158+
#.(BENCHMARKRESULTS REC-FIB-GENERIC2 1002 911697 910 #.(SOME 22.68230092439423d0) #.(SOME 0))
159+
#.(BENCHMARKRESULTS REC-FIB-GENERIC3 1014 10270171 10128 #.(SOME 318.8679543261109d0) #.(SOME 0))
160+
#.(BENCHMARKRESULTS REC-FIB-LISP 1000 104998 105 #.NONE #.(SOME 0))
161+
#.(BENCHMARKRESULTS REC-FIB-MONO 1000 306261 306 #.NONE #.(SOME 0))))
106162
```
107163

108164
`#:run-benchmarks` runs the current package's benchmarks.

0 commit comments

Comments
 (0)