Update benchmarks

.github/workflows/ci.yaml

@@ -315,6 +315,16 @@ jobs:
           git diff --ignore-cr-at-eol --exit-code \
             unison-src/transcripts-manual/dll-ffi-win.output.md
 
+      - name: benchmark smoke tests
+        # benchmarks hang on Windows
+        if: steps.cache-transcript-test-results.outputs.cache-hit != 'true' && runner.os != 'Windows'
+        run: |
+          ${{env.ucm}} transcript unison-src/transcripts-manual/benchmarks.md
+
+          # Fail if the output or generated docs differ.
+          git diff --ignore-cr-at-eol --exit-code \
+            unison-src/transcripts-manual/benchmarks.md
+
       - name: mark transcripts as passing
         if: steps.cache-transcript-test-results.outputs.cache-hit != 'true'
         run: |

development.markdown

@@ -50,7 +50,7 @@ Some tests are executables instead:
 * `stack exec transcripts -- prefix-of-filename` runs only transcript tests with a matching filename prefix.
 * `stack exec cli-integration-tests` runs the additional integration tests for cli. These tests are not triggered by `tests` or `transcripts`.
 * `stack exec unison -- transcript unison-src/transcripts-round-trip/main.md` runs the pretty-printing round trip tests
-* `stack exec unison -- transcript unison-src/transcripts-manual/benchmarks.md` runs the benchmark suite. Output goes in unison-src/transcripts-manual/benchmarks/output.txt.
+* `sudo ./scripts/benchmark` runs the benchmark suite. (If you can’t or won’t run under `sudo` it will just run with lower scheduling priority)
 
 ### Building everything at once, including tests and benchmarks, but without running them:
 Do:

scripts/benchmark

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+set -euo pipefail
+SCRIPTDIR="$(dirname -- "$(readlink -f -- "${BASH_SOURCE[0]}")")"
+
+### Run the Unison benchmark suite.
+###
+### NB: If run as a superuser (e.g., under `sudo`), this will use a higher priority to hopefully get more stable
+###     results.
+
+## TODO: `transcripts` should be able to take a search path, but since it currently looks for files relative to its run
+##       path, we `cd` to where it can find them.
+cd "$SCRIPTDIR/.."
+
+niceness=0
+if [[ $EUID -eq 0 ]]; then
+    niceness=-20
+fi
+
+nice -n $niceness stack exec unison -- transcript unison-src/transcripts-manual/benchmarks.md

unison-src/transcripts-manual/benchmarks.md

@@ -1,96 +1,7 @@
 ``` ucm :hide
-> pull unison.public.base.releases.M4d base
-> pull runarorama.public.sort.data sort
-```
-
-``` unison :hide
-benchmarkFilePath = FilePath "unison-src/transcripts-manual/benchmarks/output.bench.txt"
-archiveFilePath = FilePath "unison-src/transcripts-manual/benchmarks/output"
-
-timeit : Text -> '{IO,Exception} a ->{IO,Exception} a
-timeit label a =
-  before = !realtime
-  r = !a
-  after = !realtime
-  elapsed = Duration.between before after
-  elapsedText = Duration.toText elapsed
-  go file =
-    putText file ("\n" ++ label ++ " " ++ Int.toText (Duration.countMicroseconds elapsed) ++ "   # " ++ elapsedText)
-    printLine ("\n\n ******** \n")
-    printLine (label ++ " took " ++ elapsedText)
-  bracket '(FilePath.open benchmarkFilePath FileMode.Append) Handle.close go
-  r
-
-prepare = do
-  -- if benchmarkFilePath exists, move it to blah-<datetime>.txt for archive purposes
-  use Text ++
-  if FilePath.exists benchmarkFilePath then
-    createDirectory archiveFilePath
-    now = OffsetDateTime.toText (atUTC !realtime)
-    timestamped = FilePath.toText archiveFilePath ++ "/" ++ now ++ "-bench.txt"
-    renameFile benchmarkFilePath (FilePath timestamped)
-  else
-    ()
-```
-
-``` ucm :hide
-> add
-> run prepare
-```
-
-## Benchmarks
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/each.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/listmap.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/listfilter.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/random.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/simpleloop.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/fibonacci.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/map.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/natmap.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/stm.u
-> run main
-```
-
-``` ucm
-> load unison-src/transcripts-manual/benchmarks/tmap.u
-> run main
+> pull @pchiusano/misc-benchmarks
 ```
 
 ``` ucm
-> load unison-src/transcripts-manual/benchmarks/array-sort.u
-> run main
+> run suite
 ```

unison-src/transcripts-manual/benchmarks.output.md

@@ -0,0 +1,9 @@
+``` ucm :hide
+> pull @pchiusano/misc-benchmarks
+```
+
+``` ucm
+> run suite
+
+  ()
+```