-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
433 lines (400 loc) · 23.5 KB
/
index.html
File metadata and controls
433 lines (400 loc) · 23.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="Sergey Mechtaev">
<title>Modus</title>
<link rel="stylesheet" href="assets/dist/css/hljs-vs.css">
<!-- Bootstrap core CSS -->
<link href="assets/dist/css/bootstrap.min.css" rel="stylesheet">
<!-- Place this tag in your head or just before your close body tag. -->
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.8.1/font/bootstrap-icons.css">
<style>
.bd-placeholder-img {
font-size: 1.125rem;
text-anchor: middle;
-webkit-user-select: none;
-moz-user-select: none;
user-select: none;
}
@media (min-width: 768px) {
.bd-placeholder-img-lg {
font-size: 3.5rem;
}
}
pre[class^="language-"] {
padding: 0.5em;
/* margin: .5em 0; */
overflow: auto;
border: 1px solid #dddddd;
/* background-color: white; */
}
.hljs {
background-color: transparent !important;
}
</style>
<!-- Custom styles for this template -->
<link href="modus.css" rel="stylesheet">
</head>
<body class="pb-0 mb-0">
<nav class="navbar navbar-expand-lg navbar-light modus-bg mb-0" aria-label="Modus navbar">
<div class="container">
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#modusNavbar"
aria-controls="navbarsExample07" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="modusNavbar">
<ul class="navbar-nav me-auto mb-2 mb-lg-0">
<li class="nav-item">
<a class="navbar-brand" href="/">
<img src="assets/images/modus.svg" width="35" alt="Modus logo">
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://docs.modus-continens.com">Docs</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://github.com/modus-continens/modus">GitHub</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://github.com/modus-continens/modus/issues">Issue Tracker</a>
</li>
<li class="nav-item">
<a class="nav-link" href="https://discord.gg/bXxwfVE9Kj">Discord</a>
</li>
<li>
<a class="nav-link" href="https://play.modus-continens.com">Playground</a>
</li>
</ul>
<a class="btn btn-outline-secondary mt-0" href="https://discord.gg/bXxwfVE9Kj" role="button"><i class="bi bi-people"></i> Join Community</a>
</div>
</div>
</nav>
<div class="modus-bg jumbotron">
<div class="p-5">
<div class="container">
<div>
<div class="card bg-transparent border-0">
<div class="row g-0">
<img class="modus-logo-img" src="assets/images/modus.svg" alt="Modus logo">
<div class="col-md-8">
<div class="card-body pt-0 mt-0">
<h1 class="pb-0 mb-0 mt-0 pt-4">Modus</h1>
<p class="">A language for building Docker/OCI container images</p>
<a class="github-button" href="https://github.com/modus-continens/modus" data-icon="octicon-star"
data-size="large" data-show-count="true" aria-label="Star modus-continens/modus on GitHub">Star</a>
</div>
</div>
</div>
</div>
<!-- Place this tag where you want the button to render. -->
<div class="row mt-2">
<div class="col-lg-8 col-md-12">
<p class="lead">Modus uses logic programming to express interactions among build parameters, specify complex build workflows, automatically parallelise and cache builds, help to reduce image size, and simplify maintenance.</p>
<!-- Place this tag where you want the button to render. -->
<p>
<a class="btn btn-primary mt-0" href="get.html"
role="button"><i class="bi bi-download"></i> Install Modus</a>
<a class="btn btn-info mt-0" href="https://docs.modus-continens.com/tutorial.html"
role="button"><i class="bi bi-book"></i> Read Tutorial</a>
</p>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="bg-light p-5">
<div class="container">
<div class="row gx-5">
<div class="col-lg-6 col-md-12">
<p class="lead">A Modus program is a set of rules that define how new images are built from existing images by adding filesystem layers. Images and layers are represented using predicates, which may have parameters. When building an image, Modus automatically resolves parameter values and computes a build DAG containing all the instructions necessary to construct the image. Modus automatically caches these instructions and executes them in parallel.</p>
</div>
<div class="col-lg-6 col-md-12">
<p>This Modusfile defines the image <code>my_app</code> with the parameter <code>profile</code>. Depending on the value of <code>profile</code>, it builds either a debug or a release binary. The operators <code>::set_workdir</code> and <code>::set_entrypoint</code> set image properties:
<pre><code class="language-modusfile">
my_app(profile) :-
(
from("rust:alpine")::set_workdir("/usr/src/app"),
copy(".", "."),
cargo_build(profile)
)::set_entrypoint(f"./target/${profile}/my_app").
cargo_build("debug") :- run("cargo build").
cargo_build("release") :- run("cargo build --release").</code></pre>
</div>
</div>
</div>
</div>
<div class="p-5">
<div class="container">
<h2 class="featurette-heading" id="parameterised-builds">Dockerfiles vs Modus</h2>
<table class="table table-striped">
<thead>
<tr>
<th scope="col"></th>
<th scope="col">Dockerfiles</th>
<th scope="col">Modus</th>
</tr>
</thead>
<tbody>
<tr>
<th scope="row">Parameter Interaction</th>
<td>Do not track dependencies among build parameters.</td>
<td>Tracks and automatically resolves dependencies among build parameters.</td>
</tr>
<tr>
<th scope="row">Parallelisation</th>
<td>Support custom workflows only by resorting to scripts which inefficiently parallelise.</td>
<td>Aggressively parallelises builds involving custom logic.</td>
</tr>
<tr>
<th scope="row">Caching</th>
<td>Ineffectively cache custom workflows expressed as embedded scripts.</td>
<td>Provides effective caching for custom workflows.</td>
</tr>
<tr>
<th scope="row">Image size</th>
<td>Tend to produce both redundant layers and layers with more files and packages than required.</td>
<td>Avoids redundancies via its precise dependencies encoding and permits merging unnecessary layers.</td>
</tr>
<tr>
<th scope="row">Maintainability</th>
<td>Rely on hard-coded configuration and lack code reuse, so they're hard to maintain.</td>
<td>Provides zero-cost modularity and code reuse, so Modusfiles are easy to maintain.</td>
</tr>
</tbody>
</table>
</div>
</div>
<div>
<div class="bg-light p-5">
<div class="container">
<h2 class="featurette-heading" id="parameterised-builds">Build Parameter Dependencies</h2>
<div class="row gx-5">
<div class="col-lg-6 col-md-12">
<p class="lead">Container images are intrinsically parameterised, e.g. <code>python:3.9-slim-bullseye</code> is parameterised with Python's version <code>3.9</code> and Debian's options <code>slim</code> and <code>bullseye</code>. These parameters can, and often do, depend on and interact with each other and these interactions determine how images are built. Dockerfiles only support parameters as global variables, and do not handle dependencies between them. Developers either <a href="https://github.com/klee/klee/blob/0ba95edbad26fe70c8132f0731778d94f9609874/Dockerfile#L1-L34">hard-code version dependencies</a> or implement ad-hoc Dockerfile generators. For example, Official OpenJDK Docker Images use a combination of <a href="https://github.com/docker-library/openjdk/blob/20e86dbd02a19bca2f66b46bc3e8b00170f6f69c/Dockerfile-linux.template">Dockerfiles templates</a> with embedded JQ queries, <a href="https://github.com/docker-library/bashbrew/blob/1da7341a79651d28fbcc3d14b9176593c4231942/scripts/jq-template.awk">AWK scripts</a> and <a href="https://github.com/docker-library/openjdk/blob/abebf9325fea4606b9759fb3b9257ea3eef40061/apply-templates.sh">Bash scripts</a> to support parametrisation.</p>
<p class="lead">Modus capitalises on its <a href="https://docs.modus-continens.com/foundations">logic programming foundation</a> to handle parameters and their dependencies in an intuitive, declarative fashion. Modus decreased the size of OpenJDK Docker images build system by 47.6% from scripts written in three languages to a single Modusfile, while reducing the build time by 40.6%.</p> <p><a class="lead" href="https://github.com/modus-continens/openjdk-images-case-study">OpenJDK images case study <i class="bi bi-arrow-right-circle"></i></a></p>
</div>
<div class="col-lg-6 col-md-12">
<p>This fragment of <a href="https://github.com/docker-library/openjdk/blob/20e86dbd02a19bca2f66b46bc3e8b00170f6f69c/Dockerfile-linux.template">OpenJDK Dockerfile template</a> combines Dockerfile with two external tools: (1) <code>{{</code> syntax handled by an AWK script and (2) predicates expressed as JQ queries:
<pre><code class="language-dockerfile">
FROM {{
if is_debian_slim then
"debian:" + debian_suite + "-slim"
else
"buildpack-deps:" + debian_suite + (
if env.javaType == "jdk" then
"-scm"
else
"-curl"
end
)
end
}}
</code></pre>
<p>An <a href="https://github.com/modus-continens/openjdk-images-case-study/blob/main/linux.Modusfile">equivalent Modusfile</a> expresses this fragment without external tools:</p>
<pre><code class="language-modusfile">
debian_image(VARIANT, JAVA_TYPE) :-
(
is_debian_slim(VARIANT, DEBIAN_SUITE),
from(f"debian:${DEBIAN_SUITE}-slim")
;
is_debian(VARIANT),
debian_suffix_type(SUFFIX, JAVA_TYPE),
from(f"buildpack-deps:${VARIANT}${SUFFIX}")
).
debian_suffix_type("-scm", "jdk").
debian_suffix_type("-curl", "jre").
</code></pre>
</p>
</div>
</div>
</div>
</div>
<div class="p-5">
<div class="container">
<div class="row gx-5">
<h2 class="featurette-heading" id="parallel-builds">Parallel Builds</h2>
<div class="col-lg-6 col-md-12">
<p class="lead">Building container images is time-consuming. A single Dockerfile effectively captures a linear workflow and can be effectively paralellised. Custom workflows, however, require augmenting Dockerfiles with templates and scripts and the combination can be hard to parallelise. For example, the <a href="https://github.com/docker-library/openjdk/blob/20e86dbd02a19bca2f66b46bc3e8b00170f6f69c/Dockerfile-linux.template">OpenJDK Dockerfile template</a> builds OpenJDK images 40.6% slower than Modus, because the templating scripts run sequentially. Alternative solutions, such as <a href="https://github.com/containers/buildah">Buildah scripts</a>, are also difficult to automatically parallelise.</p>
<p class="lead">Modus statically constructs the build graph consisting of all required operations to build target images, which enables it to aggressively paralellise build with <a href="https://github.com/moby/buildkit">BuildKit</a>. When building multiple images in parallel, Modus reuses shared layers across them.</p>
</div>
<div class="col-lg-6 col-md-12">
<p>The query <code>openjdk(version, "jdk", variant), number_gt(version, 11)</code> for <a href="https://github.com/modus-continens/openjdk-images-case-study">OpenJDK Modusfile</a> builds all variants of all JDK versions greater than 11. Modus builds 23 images in parallel, reusing intermediate layers across images:</p>
<pre><code class="language-shell">
$ modus build . 'openjdk(version, "jdk", variant), number_gt(version, 11)'
Exporting 1/23: openjdk("17", "jdk", "bullseye") ->
sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248
Exporting 2/23: openjdk("17", "jdk", "slim-buster") ->
sha256:c34ce3c1fcc0c7431e1392cc3abd0dfe2192ffea1898d5250f199d3ac8d8720f
...
Exporting 23/23: openjdk("18", "jdk", "buster") ->
sha256:220611111e8c9bbe242e9dc1367c0fa89eef83f26203ee3f7c3764046e02b248
</code></pre>
</div>
</div>
</div>
</div>
<div class="bg-light p-5">
<div class="container">
<div class="row gx-5">
<h2 class="featurette-heading" id="parallel-builds">Caching</h2>
<div class="col-lg-6 col-md-12">
<p class="lead">Caching is crucial for optimising build time, because it reuses previously built layers when there are no changes in the build environment and configuration. Caching is only effective when the build system correctly detects cache invalidation. It is a common practice to implement custom build logic by embedding scripts into Dockerfile's <code>RUN</code> instructions. Embedded scripts make cache invalidation imprecise, since any change of the script invalidates the cache even if it is irrelevant to the build target.</p>
<p class="lead">Modus statically computes the exact sequence of instructions required to build the target image. This enables users to describe custom logic without sacrificing automatic caching, since Modus does not invalidate cache when irrelevant parts of build logic are modified.</p>
</div>
<div class="col-lg-6 col-md-12">
<p>In this Dockerfile, the build profile is selected by an embedded shell script using the argument <code>PROFILE</code>. The build cache for the target <code>PROFILE=debug</code> will be invalidated even if only the irrelevant line <code>make program ; \</code> is modified:</p>
<pre><code class="language-dockerfile">
FROM gcc:bullseye AS app
COPY program.c program.c
ARG PROFILE
RUN if [ "$PROFILE" = "debug" ] ; then \
CFLAGS=-g make -e program ; \
else \
make program ; \
fi
</code></pre>
<p>In the equivalent Modusfile, cache invalidation depends on the exact executed commands, therefore a change of the body of <code>make("release")</code> will not invalidate the cache of <code>app("debug")</code>:</p>
<pre><code class="language-modusfile">
app(profile) :-
from("gcc:bullseye"),
copy("program.c", "program.c"),
make(profile).
make("debug") :- run("make -e program")::in_env("CFLAGS", "-g").
make("release") :- run("make program").
</code></pre>
</div>
</div>
</div>
</div>
<div class="p-5">
<div class="container">
<div class="row gx-5">
<h2 class="featurette-heading" id="optimising-image-size">Optimising Image Size</h2>
<div class="col-lg-6 col-md-12">
<p class="lead">Container images often include redundant layers, files and installed packages, which greatly increases their size, slows down their transfer through network, and compromises security by increasing the attack surface. At Stackoverflow, the question <a href="https://stackoverflow.com/questions/24394243/why-are-docker-container-images-so-large"> Why are Docker container images so large?</a> has 103k views. Dockerfiles cannot conditionally install packages and copy files based on the build configuration without sacrificing caching and parallelism, and do not provide tool for fine-grained control of layers.</p>
<p class="lead">Modus provides predicates and operators for querying and modifying the build environment. Together, they allow the user to precisely define the files and software packages their build configuration requires. Modus provides constructs to reduce image size such as the operator <code>::merge</code> that merges several layers into one, and the operator <code>::copy</code> for conveniently defining <a href="https://docs.docker.com/develop/develop-images/multistage-build/">multi-stage builds</a>.</p>
</div>
<div class="col-lg-6 col-md-12">
<p>The operator <code>::merge</code> is applied to a fragment of code to ensure that it will produce a single layers. As a result, the directory <code>src</code> will not be stored in an intermediate layer:</p>
<pre><code class="language-modusfile">
app(build_mode) :-
from("gcc:latest"),
(
copy("src", "src"),
make(build_mode),
run("rm -rf src")
)::merge.
make("release") :- run("cd src; make install").
make("debug") :- run("cd src; make -e install")::in_env("CFLAGS", "-g").</code></pre>
<p>The operator <code>::copy</code> is applied to copy a file converted to UNIX format from a temporary image, without requiring installation of the package <code>dos2unix</code> on the target image <code>app</code>:</p>
<pre><code class="language-modusfile">
copy_convert(file, dest) :-
(
from("debian:bullseye-slim"),
run("apt-get update && apt-get install dos2unix"),
copy(file, f"/tmp/${file}"),
run(f"dos2unix /tmp/${file}")
)::copy(f"/tmp/${file}", dest).
app :-
from("debian:bullseye-slim"),
copy_convert("my_local_script.sh", ".").</code></pre>
</div>
</div>
</div>
</div>
<div class="bg-light p-5">
<div class="container">
<div class="row gx-5">
<h2 class="featurette-heading" id="modularity-and-code-reuse">Modularity & Code Reuse</h2>
<div class="col-lg-6 col-md-12">
<p class="lead">Just like any code, container build definitions evolve and require maintenance. Dockerfiles do not provide features for modularity and code reuse. Besides, to optimise image size, they require structuring code in an <a href="https://nickjanetakis.com/blog/docker-tip-3-chain-your-docker-run-instructions-to-shrink-your-images">way</a> that <a href="https://github.com/moby/moby/issues/16058">many dislike</a>.</p>
<p class="lead">Modus supports code evolution and maintenance by providing zero-cost modularity and code reuse. Modus allows users to define their own commands, such as layer building functions or logical predicates, to abstract reusable build workflows. Modus provides a library of builtin predicates to handle common data structures in the build logic. For example, the predicate <code>semver_geq</code> checks if the left version is greater or equal to the right version according to <a href="https://semver.org/">SemVer</a> specification.</p>
<p><a class="lead" href="https://docs.modus-continens.com/library/predicates/index.html">List of builtin predicates <i class="bi bi-arrow-right-circle"></i></a></p>
<p class="lead">Modus provides a library of operators, such as <code>::in_env</code> for executing commands in a custom environment, that encapsulate build-specific instructions and manipulation of OCI image properties.</p>
<p><a class="lead" href="https://docs.modus-continens.com/library/operators/index.html">List of operators <i class="bi bi-arrow-right-circle"></i></a></p>
</div>
<div class="col-lg-6 col-md-12">
<p>Using a user-defined predicate <code>install</code> to reuse library installation code:</p>
<pre><code class="language-modusfile">
install(lib, version) :-
run(f"wget https://example.com/libs/${lib}-v${version}.tar.gz && \
tar xf ${lib}-v${version}.tar.gz && \
mv ${lib}-v${version}/ /build"),
run("cd /build && make install"),
run(f"rm ${lib}-v${version}.tar.gz && \
rm -rf /build").
app :-
from("gcc:latest"),
install("liba", "1.3.5"),
install("libb", "4.1").</code></pre>
<p>Using the built-in predicate <code>semver_geq</code> to compare versions of Ubuntu:</p>
<pre><code class="language-modusfile">
base(distr_version, python_version) :-
semver_geq(distr_version, "16.04"),
from(f"ubuntu:${distr_version}"),
run(f"apt-get update && apt-get install -y python${python_version} \
&& rm -rf /var/lib/apt/lists/*").
</code></pre>
<p>Using the built-in operator <code>::in_env</code> to execute commands in a custom environment:</p>
<pre><code class="language-modusfile">
app :-
from("debian:bullseye-slim"),
(
run("apt-get update"),
run("apt-get upgrade"),
run("apt-get install build-essential")
)::in_env("DEBIAN_FRONTEND", "noninteractive").</code></pre>
</div>
</div>
</div>
</div>
<div class="p-5">
<div class="container">
<div class="row gx-5">
<h2 class="featurette-heading" id="modularity-and-code-reuse">Research & Development</h2>
<div class="col-lg-6 col-md-12">
<p class="lead">Modus is a research project developed at University College London. Modus is led by <a href="https://mechtaev.com/">Dr. Sergey Mechtaev</a> and <a href="https://earlbarr.com/">Prof. Earl T. Barr</a>. Modus aims to be a practical tool that is used by both computer science researchers and software developers.</p>
<p class="lead">Research on Modus is published in peer-reviewed venues, and follows the principles of open science. All our code and data are released as reproducible packages on <a href="https://github.com/modus-continens/">GitHub</a>.</p>
</div>
<div class="col-lg-6 col-md-12">
<p class="lead">For more details, please read our FSE'22 paper on Modus:</p>
<a href="assets/files/fse22.pdf">Modus: A Datalog Dialect for Building Container Images</a><br> <em>Chris Tomy, Tingmao Wang, Earl Barr, Sergey Mechtaev</em><br>The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)
</div>
</div>
</div>
</div>
</div>
<div class="bg p-5">
<div class="container">
<footer>
<p>Modus is distributed under <a href="https://www.gnu.org/licenses/agpl-3.0.txt">GNU Affero General Public
License v3.0</a>. © 2022 University College London.</p>
</footer>
</div>
</div>
</div>
<script src="assets/dist/js/bootstrap.bundle.min.js"></script>
<script async defer src="https://buttons.github.io/buttons.js"></script>
<script src="assets/dist/js/highlight.min.js"></script>
<script>hljs.highlightAll();</script>
<script>
for (let code of document.querySelectorAll("pre > code[class^='language-']")) {
code.parentElement.className = code.className;
let f = code.childNodes[0];
// Remove empty first and last line inserted by HTML.
if (f instanceof Text) {
if (f.textContent.startsWith("\n")) {
f.textContent = f.textContent.slice(1);
}
if (/\n\s*/m.test(f.textContent)) {
f.textContent = f.textContent.replace(/\n\s*$/m, "")
}
}
}
</script>
</body>
</html>