Skip to content

Commit 6ca912f

Browse files
ErikSchierboomBNAndras
ErikSchierboom
and
BNAndras
authored
Add best practices for tooling (#551)
* Update docker * Lots of work * More best practice * Apply suggestions from code review Co-authored-by: András B Nagy <[email protected]> * Fix copy-paste error * Add run command * Fix timeout --------- Co-authored-by: András B Nagy <[email protected]>
1 parent 3d692cf commit 6ca912f

10 files changed

+320
-20
lines changed

‎building/config.json

+7
Original file line numberDiff line numberDiff line change
@@ -717,6 +717,13 @@
717717
"title": "Tooling Docker Setup",
718718
"blurb": ""
719719
},
720+
{
721+
"uuid": "83dc65b9-cc74-4a2a-9c70-472f6416c19d",
722+
"slug": "tooling/best-practices",
723+
"path": "building/tooling/best-practices.md",
724+
"title": "Best Practices",
725+
"blurb": ""
726+
},
720727
{
721728
"uuid": "b080e814-d3b9-4027-a25e-907f5505cf8d",
722729
"slug": "tooling/analyzers",

‎building/tooling/analyzers/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -21,3 +21,4 @@ You can use the following documents to learn more about building an analyzer:
2121
- [Writing Analyzer comments](/docs/building/tooling/analyzers/comments)
2222
- [Tagging solutions](/docs/building/tooling/analyzers/tags)
2323
- [Guidance for building an Analyzer](/docs/building/tooling/analyzers/guidance)
24+
- [Best practices](/docs/building/tooling/best-practices)

‎building/tooling/analyzers/creating-from-scratch.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ Firstly, thank you for your interest in creating an Analyzer!
44

55
These are the steps to get going:
66

7-
1. Check [our repository list for an existing `...-analyzer`](https://github.com/exercism?q=-analyzer) to ensure that one doesn't already exist.
7+
1. Check [our repository list for an existing `...-analyzer`](https://github.com/search?q=org%3Aexercism+analyzer&type=repositories) to ensure that one doesn't already exist.
88
2. Scan the [contents of this directory](/docs/building/tooling/analyzers) to ensure you are comfortable with the idea of creating an Analyzer.
9-
3. Open an issue at [exercism/exercism][exercism-repo] introducing yourself and telling us which language you'd like to create a Analyzer for.
9+
3. Start a new topic on [the Exercism forum][building-exercism] telling us which language you'd like to create an Analyzer for.
1010
4. Once an Analyzer repo has been created, use [the Analyzer interface document](/docs/building/tooling/analyzers/interface) to help guide your implementation.
1111

1212
We have an incredibly friendly and supportive community who will be happy to help you as you work through this! If you get stuck, please start a new topic on [the Exercism forum][building-exercism] or create new issues at [exercism/exercism][exercism-repo] as needed 🙂

‎building/tooling/best-practices.md

+291
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,291 @@
1+
# Best Practices
2+
3+
## Follow official best practices
4+
5+
The official [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) have lots of great content on how to improve your Dockerfiles.
6+
7+
## Performance
8+
9+
You should primarily optimize for performance (especially for test runners).
10+
This will ensure your tooling runs as fast as possible and does not time-out.
11+
12+
### Experiment with different Base images
13+
14+
Try experimenting with different base images (e.g. Alpine instead of Ubuntu), to see if one (significantly) outperforms the other.
15+
If performance is relatively equal, go for the image that is smallest.
16+
17+
### Try Internal Network
18+
19+
Check if using the `internal` network instead of `none` improves performance.
20+
See the [network docs](/docs/building/tooling/docker#network) for more information.
21+
22+
### Prefer build-time commands over run-time commands
23+
24+
Tooling runs as one-off, short-lived Docker container:
25+
26+
1. A Docker container is created
27+
2. The Docker container is run with the correct arguments
28+
3. The Docker container is destroyed
29+
30+
Therefore, code that runs in step 2 runs for _every single tooling run_.
31+
For this reason, reducing the amount of code that runs in step 2 is a great way to improve performance
32+
One way of doing this is to move code from _run-time_ to _build-time_.
33+
Whilst run-time code runs on every single tooling run, build-time code only runs once (when the Docker image is built).
34+
35+
Build-time code runs once as part of a GitHub Actions workflow.
36+
Therefore, its fine if the code that runs at build-time is (relatively) slow.
37+
38+
#### Example: pre-compile libraries
39+
40+
When running tests in the Haskell test runner, it requires some base libraries to be compiled.
41+
As each test run happens in a fresh container, this means that this compilation was done _in every single test run_!
42+
To circumvent this, the [Haskell test runner's Dockerfile](https://github.com/exercism/haskell-test-runner/blob/5264c460054649fc672c3d5932c2f3cb082e2405/Dockerfile) has the following two commands:
43+
44+
```dockerfile
45+
COPY pre-compiled/ .
46+
RUN stack build --resolver lts-20.18 --no-terminal --test --no-run-tests
47+
```
48+
49+
First, the `pre-compiled` directory is copied into the image.
50+
This directory is setup as a sort of fake exercise and depends on the same base libraries that the actual exercise depend on.
51+
Then we run the tests on that directory, which is similar to how tests are run for an actual exercise.
52+
Running the tests will result in the base being compiled, but the difference is that this happens at _build time_.
53+
The resulting Docker image will thus have its base libraries already compiled, which means that no longer has to happen at _run time_, resulting in (much) faster execution times.
54+
55+
#### Example: pre-compile binaries
56+
57+
Some languages allow code to be compiled ahead-of-time or just-in-time.
58+
This is a build time vs. run time tradeoff, and again, we favor build time execution for performance reasons.
59+
60+
The [C# test runner's Dockerfile](https://github.com/exercism/csharp-test-runner/blob/b54122ef76cbf86eff0691daa33c8e50bc83979f/Dockerfile) uses this approach, where the test runner is compiled to a binary ahead-of-time (at build time) instead of just-in-time compiling the code (at run time).
61+
This means that there is less work to do at run-time, which should help increase performance.
62+
63+
## Size
64+
65+
You should try to reduce the image's size, which means that it'll:
66+
67+
- Be faster to deploy
68+
- Reduce costs for us
69+
- Improve startup time of each container
70+
71+
### Try different distributions
72+
73+
Different distribution images will have different sizes.
74+
For example, the `alpine:3.20.2` image is **ten times** smaller than the `ubuntu:24.10` image:
75+
76+
```
77+
REPOSITORY TAG SIZE
78+
alpine 3.20.2 8.83MB
79+
ubuntu 24.10 101MB
80+
```
81+
82+
In general, Alpine-based images are amongst the smallest images, so many tooling images are based on Alpine.
83+
84+
### Try slimmed-down images
85+
86+
Some images have special "slim" variants, in which some features will have been removed resulting in smaller image sizes.
87+
For example, the `node:20.16.0-slim` image is **five times** smaller than the `node:20.16.0` image:
88+
89+
```
90+
REPOSITORY TAG SIZE
91+
node 20.16.0 1.09GB
92+
node 20.16.0-slim 219MB
93+
```
94+
95+
The reason "slim" variants are smaller is that they'll have less features.
96+
Your image might not need the additional features, and if not, consider using the "slim" variant.
97+
98+
### Removing unneeded bits
99+
100+
An obvious, but great, way to reduce the size of your image is to remove anything you don't need.
101+
These can include things like:
102+
103+
- Source files that are no longer needed after building a binary from them
104+
- Files targeting different architectures from the Docker image
105+
- Documentation
106+
107+
#### Remove package manager files
108+
109+
Most Docker images need to install additional packages, which is usually done via a package manager.
110+
These packages must be installed at _build time_ (as no internet connection is available at _run time_).
111+
Therefore, any package manager caching/bookkeeping files should be removed after installing the additional packages.
112+
113+
##### apk
114+
115+
Distributions that uses the `apk` package manager (such as Alpine) should use the `--no-cache` flag when using `apk add` to install packages:
116+
117+
```dockerfile
118+
RUN apk add --no-cache curl
119+
```
120+
121+
##### apt-get/apt
122+
123+
Distributions that use the `apt-get`/`apk` package manager (such as Ubuntu) should run the `apt-get autoremove -y` and `rm -rf /var/lib/apt/lists/*` commands _after_ installing the packages and in the same `RUN` command:
124+
125+
```dockerfile
126+
RUN apt-get update && \
127+
apt-get install curl -y && \
128+
apt-get autoremove -y && \
129+
rm -rf /var/lib/apt/lists/*
130+
```
131+
132+
### Use multi-stage builds
133+
134+
Docker has a feature called [multi-stage builds](https://docs.docker.com/build/building/multi-stage/).
135+
These allow you to partition your Dockerfile into separate _stages_, with only the last stage ending up in the produced Docker image (the rest is only there to support building the last stage).
136+
You can think of each stage as its own mini Dockerfile; stages can use different base images.
137+
138+
Multi-stage builds are particularly useful when your Dockerfile requires packages to be installed that are _only_ needed at build time.
139+
In this situation, the general structure of your Dockerfile looks like this:
140+
141+
1. Define a new stage (we'll call this the "build" stage).
142+
This stage will _only_ be used at build time.
143+
2. Install the required additional packages (into the "build" stage).
144+
3. Run the commands that require the additional packages (within the "build" stage).
145+
4. Define a new stage (we'll call this the "runtime" stage).
146+
This stage will make up the resulting Docker image and executed at run time.
147+
5. Copy the result(s) from the commands run in step 3 (in the "build" stage) into this stage (the "runtime" stage).
148+
149+
With this setup, the additional packages are _only_ installed in the "build" stage and _not_ in the "runtime" stage, which means that they won't end up in the Docker image that is produced.
150+
151+
#### Example: downloading files
152+
153+
The Fortran test runner requires `curl` to download some files.
154+
However, its run time image does _not_ need `curl`, which makes this a perfect use case for a multi-stage build.
155+
156+
First, its [Dockerfile](https://github.com/exercism/fortran-test-runner/blob/783e228d8449143d2040e68b95128bb791833a27/Dockerfile) defines a stage (named "build") in which the `curl` package is installed.
157+
It then uses curl to download files into that stage.
158+
159+
```dockerfile
160+
FROM alpine:3.15 AS build
161+
162+
RUN apk add --no-cache curl
163+
164+
WORKDIR /opt/test-runner
165+
COPY bust_cache .
166+
167+
WORKDIR /opt/test-runner/testlib
168+
RUN curl -R -O https://raw.githubusercontent.com/exercism/fortran/main/testlib/CMakeLists.txt
169+
RUN curl -R -O https://raw.githubusercontent.com/exercism/fortran/main/testlib/TesterMain.f90
170+
171+
WORKDIR /opt/test-runner
172+
RUN curl -R -O https://raw.githubusercontent.com/exercism/fortran/main/config/CMakeLists.txt
173+
```
174+
175+
The second part of the Dockerfile defines a new stage and copies the downloaded files from the "build" stage into its own stage using the `COPY` command:
176+
177+
```dockerfile
178+
FROM alpine:3.15
179+
180+
RUN apk add --no-cache coreutils jq gfortran libc-dev cmake make
181+
182+
WORKDIR /opt/test-runner
183+
COPY --from=build /opt/test-runner/ .
184+
185+
COPY . .
186+
ENTRYPOINT ["/opt/test-runner/bin/run.sh"]
187+
```
188+
189+
##### Example: installing libraries
190+
191+
The Ruby test runner needs the `git`, `openssh`, `build-base`, `gcc` and `wget` packages to be installed before its required libraries (gems) can be installed.
192+
Its [Dockerfile](https://github.com/exercism/ruby-test-runner/blob/e57ed45b553d6c6411faeea55efa3a4754d1cdbf/Dockerfile) starts with a stage (given the name `build`) that install those packages (via `apk add`) and then installs the libaries (via `bundle install`):
193+
194+
```dockerfile
195+
FROM ruby:3.2.2-alpine3.18 AS build
196+
197+
RUN apk update && apk upgrade && \
198+
apk add --no-cache git openssh build-base gcc wget git
199+
200+
COPY Gemfile Gemfile.lock .
201+
202+
RUN gem install bundler:2.4.18 && \
203+
bundle config set without 'development test' && \
204+
bundle install
205+
```
206+
207+
It then defines the stage that will form the resulting Docker image.
208+
This stage does _not_ install the dependencies the previous stage installed, instead it uses the `COPY` command to copy the installed libraries from the build stage into its own stage:
209+
210+
```dockerfile
211+
FROM ruby:3.2.2-alpine3.18
212+
213+
RUN apk add --no-cache bash
214+
215+
WORKDIR /opt/test-runner
216+
217+
COPY --from=build /usr/local/bundle /usr/local/bundle
218+
219+
COPY . .
220+
221+
ENTRYPOINT [ "sh", "/opt/test-runner/bin/run.sh" ]
222+
```
223+
224+
```exercism/note
225+
The [C# test runner's Dockerfile](https://github.com/exercism/csharp-test-runner/blob/b54122ef76cbf86eff0691daa33c8e50bc83979f/Dockerfile) does something similar, only in this case the build stage can use an existing Docker image that has pre-installed the additional packages required to install libraries.
226+
```
227+
228+
## Safety
229+
230+
Safety is a main reason why we're using Docker containers to run our tooling.
231+
232+
### Prefer official images
233+
234+
There are many Docker images on [Docker Hub](https://hub.docker.com/), but try to use [official ones](https://hub.docker.com/search?q=&image_filter=official).
235+
These images are curated and have (far) less chance of being unsafe.
236+
237+
### Pin versions
238+
239+
To ensure that builds are stable (i.e. they don't suddenly break), you should always pin your base images to specific tags.
240+
That means instead of:
241+
242+
```dockerfile
243+
FROM alpine:latest
244+
```
245+
246+
you should use:
247+
248+
```dockerfile
249+
FROM alpine:3.20.2
250+
```
251+
252+
With the latter, builds will always use the same version.
253+
254+
### Run as a non-privileged user
255+
256+
By default, many images will run with a user that has root privileges.
257+
You should consider running as a non-privileged user.
258+
259+
```dockerfile
260+
FROM alpine
261+
262+
RUN groupadd -r myuser && useradd -r -g myuser myuser
263+
264+
# <RUN COMMANDS THAT REQUIRES ROOT USER, LIKE INSTALLING PACKAGES ETC.>
265+
266+
USER myuser
267+
```
268+
269+
### Update package repositories to latest version
270+
271+
It is (almost) always a good idea to install the latest versions
272+
273+
```dockerfile
274+
RUN apt-get update && \
275+
apt-get install curl
276+
```
277+
278+
### Support read-only filesystem
279+
280+
We encourage Docker files to be written using a read-only filesystem.
281+
The only directories you should assume to be writeable are:
282+
283+
- The solution dir (passed in as the second argument)
284+
- The output dir (passed in as the third argument)
285+
- The `/tmp` dir
286+
287+
```exercism/caution
288+
Our production environment currently does _not_ enforce a read-only filesystem, but we might in the future.
289+
For this reason, the base template for a new test runner/analyzer/representer starts out with a read-only filesystem.
290+
If you can't get things working on a read-only file, feel free to (for now) assume a writeable file system.
291+
```

‎building/tooling/docker.md

+11-12
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,32 @@
1-
# Tooling Docker Setup
1+
# Docker Setup
22

33
Our various track tooling are deployed as Docker images.
44

5-
Each piece of tooling requires a Dockerfile, which specifies how the machine is built.
5+
Each piece of tooling requires a [Dockerfile](https://docs.docker.com/reference/dockerfile/), which specifies how the machine is built.
66
It should live at the root directory of your repository and should be called `Dockerfile`.
77

88
The Dockerfile should create the minimal image needed for the tooling to function correctly and speedily.
9-
10-
The Dockerfile should produce an image with as a small a size as possible while maximizing (and prioritizing) performance.
11-
Applying the official [Dockerfile best practices](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/) can help to create a minimal image.
9+
Our [Best Practices page](/docs/building/tooling/best-practices) has lots of tips to help you achieve this goal.
1210

1311
## Execution
1412

1513
### Timeouts
1614

17-
Each tooling run has a ten-second window in which to execute.
18-
At the end of that period it will be timed out with a 408 error code.
15+
The test runner gets 100% CPU with 3GB of memory for a 20 second window per solution.
16+
After 20 seconds, the process is halted and reports a time-out with a 408 error code.
1917

2018
### Stdout/stderr
2119

2220
A tooling run may produce up to a maximum of one-megabyte of stdout and stderr.
23-
If it produces more it will be killed with a 413 error code.
21+
If it produces more, it will be killed with a 413 error code.
2422

2523
The contents of `stdout` and `stderr` from each run will be stored in files that can be viewed later.
2624

27-
You may write an `results.out` file to the output directory, which contains debugging information you want to later view.
25+
You may write a `results.out` file to the output directory, which contains debugging information you want to view later.
2826

2927
### Results
3028

31-
The results file may be no larger than 500 kilobytes (including any stack traces etc).
29+
The results file may not be larger than 500 kilobytes (including any stack traces etc).
3230
If the file is larger than this, the tooling run will be killed with a 460 error code.
3331

3432
## Configuration
@@ -37,7 +35,8 @@ Each solution gets 100% machine resources for a twenty second window.
3735

3836
After 20 seconds, the process is halted and reports as a time-out.
3937

40-
Configuration can be set in the [`tools.json` file](https://github.com/exercism/tooling-invoker/blob/main/tools.json) in the Tooling Invoker repository.
38+
Some tools require (slight) deviations from the default configuration.
39+
If so, these are configured in the [`tools.json` file](https://github.com/exercism/tooling-invoker/blob/main/tools.json) in the Tooling Invoker repository.
4140

4241
### Network
4342

@@ -67,4 +66,4 @@ docker container run -v /path/to/job:/mnt/exercism-iteration --network none -m 1
6766

6867
## Editing a Dockerfile
6968

70-
All changes to Dockerfiles require a PR review from the @exercism/ops team, in order to avoid the introduction of security exploits.
69+
All changes to Dockerfiles require a PR review from the `@exercism/maintainers-admin` team, in order to avoid the introduction of security exploits.

‎building/tooling/representers/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -35,3 +35,4 @@ You can use the following documents to learn more about building a representer:
3535
- [The Representer interface](/docs/building/tooling/representers/interface)
3636
- [How to normalize representations for the highest efficiency](/docs/building/tooling/representers/normalization)
3737
- [How to build a Docker image with Docker for local testing and deployment](/docs/building/tooling/representers/docker)
38+
- [Best practices](/docs/building/tooling/best-practices)

‎building/tooling/representers/creating-from-scratch.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,9 @@ Firstly, thank you for your interest in creating a Representer!
44

55
These are the steps to get going:
66

7-
1. Check [our repository list for an existing `...-representer`](https://github.com/exercism?q=-representer) to ensure that one doesn't already exist.
7+
1. Check [our repository list for an existing `...-representer`](https://github.com/search?q=org%3Aexercism+representer&type=repositories) to ensure that one doesn't already exist.
88
2. Scan the [contents of this directory](/docs/building/tooling/representers) to ensure you are comfortable with the idea of creating an Representer.
9-
3. Open an issue at [exercism/exercism][exercism-repo] introducing yourself and telling us which language you'd like to create a Representer for.
9+
3. Start a new topic on [the Exercism forum][building-exercism] telling us which language you'd like to create a Representer for.
1010
4. Once a Representer repo has been created, use [the Representer interface document](/docs/building/tooling/representers/interface) to help guide your implementation.
1111

1212
We have an incredibly friendly and supportive community who will be happy to help you as you work through this! If you get stuck, please start a new topic on [the Exercism forum][building-exercism] or create new issues at [exercism/exercism][exercism-repo] as needed 🙂

0 commit comments

Comments
 (0)