Skip to content

Commit 9ab126b

Browse files
Add measuring section to performance (#562)
1 parent dd4edbc commit 9ab126b

File tree

2 files changed

+25
-1
lines changed

2 files changed

+25
-1
lines changed

building/tooling/best-practices.md

+23
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,29 @@ The official [Dockerfile best practices](https://docs.docker.com/develop/develop
99
You should primarily optimize for performance (especially for test runners).
1010
This will ensure your tooling runs as fast as possible and does not time-out.
1111

12+
### Measure
13+
14+
Measuring execution time often is a great way to get a feel for the performance of tooling.
15+
Make it a habit to measure execution time both after _and_ before a change.
16+
Even when you feel "certain" that a change will improve performance, you should still measure execution time.
17+
18+
#### Scripts
19+
20+
When possible, create scripts to automatically measure performance (also known as _benchmarking_).
21+
A very helpful command-line tool is [hyperfine](https://github.com/sharkdp/hyperfine), but feel free to use whatever makes the most sense for your tooling.
22+
23+
Newer track tooling repos will have access to the following two scripts:
24+
25+
1. `./bin/benchmark.sh`: benchmark the track tooling code ([source code](https://github.com/exercism/generic-test-runner/blob/a6886f4d84d2a2030f766a658c334bbfbe97b79c/bin/benchmark.sh))
26+
2. `./bin/benchmark-in-docker.sh`: benchmark the track tooling Docker image ([source code](https://github.com/exercism/generic-test-runner/blob/a6886f4d84d2a2030f766a658c334bbfbe97b79c/bin/benchmark-in-docker.sh))
27+
28+
If you're working on a track tooling repo without these files, feel free to copy them into your repo.
29+
30+
```exercism/caution
31+
Benchmarking scripts can help estimate the tooling's performance.
32+
Bear in mind though that the performance on Exercism's production servers is often lower.
33+
```
34+
1235
### Experiment with different Base images
1336

1437
Try experimenting with different base images (e.g. Alpine instead of Ubuntu), to see if one (significantly) outperforms the other.

building/tooling/docker.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ Our [Best Practices page](/docs/building/tooling/best-practices) has lots of tip
1515
The test runner gets 100% CPU with 3GB of memory for a 20 second window per solution.
1616
After 20 seconds, the process is halted and reports a time-out with a 408 error code.
1717

18+
We highly recommend following our [Performance Best Practices document](/docs/building/tooling/best-practices#h-performance) to reduce the chance of timeout occuring.
19+
1820
### Stdout/stderr
1921

2022
A tooling run may produce up to a maximum of one-megabyte of stdout and stderr.
@@ -35,7 +37,6 @@ If the file is larger than this, the tooling run will be killed with a 460 error
3537
## Configuration
3638

3739
Each solution gets 100% machine resources for a twenty second window.
38-
3940
After 20 seconds, the process is halted and reports as a time-out.
4041

4142
Some tools require (slight) deviations from the default configuration.

0 commit comments

Comments
 (0)