Skip to content

Commit 8a77104

Browse files
tomowatttomowatt
authored
Merge pull request #463 from buildkite-plugins/SUP-2585-drivers
SUP-2585: Add Support for changing Build Drivers
2 parents 42fe6b5 + 8dd5ed5 commit 8a77104

File tree

7 files changed

+566
-3
lines changed

7 files changed

+566
-3
lines changed

‎README.md

+82
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,88 @@ If set to true, docker compose will build with the `--with-dependencies` option
344344

345345
The default is `false`.
346346

347+
#### `builder` (object)
348+
349+
Defines the properties required for creating, using and removing Builder Instances. If not set, the default Builder Instance on the Agent Instance will be used.
350+
351+
##### `bootstrap` (boolean)
352+
353+
If set to true, will boot builder instance after creation. Optional when using `create`.
354+
355+
The default is `true`.
356+
357+
##### `create` (boolean)
358+
359+
If set to true, will use `docker buildx create` to create a new Builder Instance using the propeties defined. If a Builder Instance with the same `name` already exists, it will not be recreated.
360+
361+
The default is `false`.
362+
363+
##### `debug` (boolean)
364+
365+
If set to true, enables debug logging during creation of builder instance. Optional when using `create`.
366+
367+
The default is `false`.
368+
369+
##### `driver`
370+
371+
If set will create a Builder Instance using the selected Driver and use it. Available Drivers:
372+
373+
- `docker-container` creates a dedicated BuildKit container using Docker.
374+
- `kubernetes` creates BuildKit pods in a Kubernetes cluster.
375+
- `remote` connects directly to a manually managed BuildKit daemon.
376+
377+
More details on different [Build Drivers](https://docs.docker.com/build/builders/drivers/).
378+
379+
##### `driver-opt`
380+
381+
Commas separated, Key-Value pairs of driver-specific options to configure the Builder Instance when using `create`. Available options for each Driver:
382+
383+
- [docker-container](https://docs.docker.com/build/builders/drivers/docker-container/)
384+
- [kubernetes](https://docs.docker.com/build/builders/drivers/kubernetes/)
385+
- [remote](https://docs.docker.com/build/builders/drivers/remote/)
386+
387+
Example: `memory=100m,cpuset-cpus=1`
388+
389+
##### `keep-daemon` (boolean)
390+
391+
If set to true, will keep the BuildKit daemon running after the buildx context (Builder) is removed. This is useful when you manage BuildKit daemons and buildx contexts independently. Only supported by the `docker-container` and `kubernetes` drivers. Optional when using `remove`.
392+
393+
The default is `false`.
394+
395+
##### `keep-state` (boolean)
396+
397+
If set to true, will keep BuildKit state so it can be reused by a new Builder with the same name, persisting the driver's cache. Currently, only supported by the `docker-container` driver. Optional when using `remove`.
398+
399+
The default is `false`.
400+
401+
##### `name`
402+
403+
Sets the name of the Builder instance to create or use. Required when using `create` or `use` builder paramaters.
404+
405+
##### `platform`
406+
407+
Commas separated, fixed platforms for builder instance. Optional when using `create`.
408+
409+
Example: `linux/amd64,linux/arm64`
410+
411+
##### `remote-address`
412+
413+
Address of remote builder instance. Required when using `driver: remote`.
414+
415+
Example: `tcp://localhost:1234`
416+
417+
##### `remove` (boolean)
418+
419+
If set to true will stop and remove the Builder Instance specified by `name`.
420+
421+
The default is `false`.
422+
423+
##### `use` (boolean)
424+
425+
If set to true will use Builder Instance specified by `name`.
426+
427+
The default is `false`.
428+
347429
## Developing
348430

349431
To run the tests:

‎docs/examples.md

+113-2
Original file line numberDiff line numberDiff line change
@@ -271,7 +271,7 @@ A newly spawned agent won't contain any of the docker caches for the first run w
271271

272272
```yaml
273273
steps:
274-
- label: ":docker Build an image"
274+
- label: ":docker: Build an image"
275275
plugins:
276276
- docker-compose#v5.4.1:
277277
build: app
@@ -295,7 +295,7 @@ The values you add in the `cache-from` will be mapped to the corresponding servi
295295

296296
```yaml
297297
steps:
298-
- label: ":docker Build an image"
298+
- label: ":docker: Build an image"
299299
plugins:
300300
- docker-compose#v5.4.1:
301301
build: app
@@ -312,3 +312,114 @@ steps:
312312
push:
313313
- app:myregistry:port/myrepo/myapp:latest
314314
```
315+
316+
### Create, Use and Remove Builder Instances
317+
318+
#### Create
319+
320+
Most Docker setups, unless configured, will use the `docker` Builder Driver by default. More details on it [here](https://docs.docker.com/build/builders/drivers/docker/).
321+
322+
The `docker` driver can handle most situations but for advance features with the Docker, different Builder Drivers are required and this requires a Builder Instance being created first, which can be done manually or with the Plugin. To create a Builder Instance using a chosen Driver, requires the `name`, `driver` and `create` parameters:
323+
324+
```yaml
325+
steps:
326+
- label: ":docker: Build an image"
327+
plugins:
328+
- docker-compose#v5.4.1:
329+
build: app
330+
push: app:index.docker.io/myorg/myrepo:my-branch
331+
cache-from:
332+
- "app:myregistry:port/myrepo/myapp:my-branch"
333+
- "app:myregistry:port/myrepo/myapp:latest"
334+
builder:
335+
name: container
336+
driver: docker-container
337+
create: true
338+
```
339+
340+
**If a Builder Instance with the same `name` already exists, it will not be recreated.**
341+
342+
#### Use
343+
344+
By default, Builder Instances specified by `name` or that are created with `create` are not used, and the default Builder Instance on the Agent will be used. To use a Builder Instance, requires the `name` and `use` parameters and the Builder Instance to exist:
345+
346+
```yaml
347+
steps:
348+
- label: ":docker: Build an image"
349+
plugins:
350+
- docker-compose#v5.4.1:
351+
build: app
352+
push: app:index.docker.io/myorg/myrepo:my-branch
353+
cache-from:
354+
- "app:myregistry:port/myrepo/myapp:my-branch"
355+
- "app:myregistry:port/myrepo/myapp:latest"
356+
builder:
357+
name: container
358+
use: true
359+
```
360+
361+
#### Remove
362+
363+
By default, Builder Instances specified by `name` or that are created with `create` are not removed after the Job finishs. To remove a Builder Instance, requires the `name` and `remove` parameters and the Builder Instance to exist:
364+
365+
```yaml
366+
steps:
367+
- label: ":docker: Build an image"
368+
plugins:
369+
- docker-compose#v5.4.1:
370+
build: app
371+
push: app:index.docker.io/myorg/myrepo:my-branch
372+
cache-from:
373+
- "app:myregistry:port/myrepo/myapp:my-branch"
374+
- "app:myregistry:port/myrepo/myapp:latest"
375+
builder:
376+
name: container
377+
driver: docker-container
378+
create: true
379+
use: true
380+
remove: true
381+
```
382+
383+
**Removing a Builder Instance by default will remove the daemon running it and its state (which can be used for cache).**
384+
**To keep the daemon or state, use the `keep-daemon` or `keep-state` parameters.**
385+
**These parameter are only applicable with specific Drivers, for detail see [`docker buildx rm`](https://docs.docker.com/reference/cli/docker/buildx/rm/).**
386+
387+
### Reusing caches from remote registries
388+
389+
A newly spawned agent won't contain any of the docker caches for the first run which will result in a long build step. To mitigate this you can reuse caches from a remote registry, but requires pushing cache and manifests to a registry using a Builder Driver that supports cache exports e.g., `docker-container` - the `docker` driver does not support this by default. For any remote registry used that requires authenication, see [Authenticated registries](#authenticated-registries) for more details. This requires use of the `cache-from`, `cache-to`, `name` and `use` parameters but will use the `create` and `driver` parameters to create the Builder Instance across multiple Agents:
390+
391+
```yaml
392+
steps:
393+
- label: ":docker: Build an image and push cache"
394+
plugins:
395+
- docker-compose#v5.4.1:
396+
build: app
397+
push: app:${DOCKER_REGISTRY}/${IMAGE_REPO}:cache
398+
cache-from:
399+
- "app:type=registry,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
400+
cache-to:
401+
- "app:type=registry,mode=max,image-manifest=true,oci-mediatypes=true,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
402+
builder:
403+
name: container
404+
use: true
405+
create: true
406+
driver: docker-container
407+
408+
- wait
409+
410+
- label: ":docker: Build an image using remote cache"
411+
plugins:
412+
- docker-compose#v5.4.1:
413+
build: app
414+
cache-from:
415+
- "app:type=registry,ref=${DOCKER_REGISTRY}/${IMAGE_REPO}:cache"
416+
builder:
417+
name: container
418+
use: true
419+
create: true
420+
driver: docker-container
421+
```
422+
423+
The first Step will build the Image using a Builder Instance with the `docker-container` driver and push the image cache to the remote registry, as specified by `cache-to`, with additional cache export options being used to export all the layers of intermediate steps with the image manifests. More details cache export options [here](https://github.com/moby/buildkit?tab=readme-ov-file#registry-push-image-and-cache-separately).
424+
425+
The second Step will build the Image using a Builder Instance with the `docker-container` driver and use remote registry for the image cache, as specified by `cache-from`, speeding up Image building process.

‎hooks/pre-command

+85
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
#!/bin/bash
2+
set -ueo pipefail
3+
4+
DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
5+
6+
# shellcheck source=lib/shared.bash
7+
. "$DIR/../lib/shared.bash"
8+
9+
builder_create="$(plugin_read_config BUILDER_CREATE "false")"
10+
builder_use="$(plugin_read_config BUILDER_USE "false")"
11+
builder_name="$(plugin_read_config BUILDER_NAME "")"
12+
13+
if [[ "${builder_create}" == "true" ]] || [[ "${builder_use}" == "true" ]]; then
14+
if [[ -z "${builder_name}" ]]; then
15+
echo "+++ 🚨 Builder Name cannot be empty when using 'create' or 'use' parameters"
16+
exit 1
17+
fi
18+
fi
19+
20+
if [[ "${builder_create}" == "true" ]]; then
21+
if ! builder_instance_exists "${builder_name}"; then
22+
builder_instance_args=(--name "${builder_name}")
23+
24+
build_driver="$(plugin_read_config BUILDER_DRIVER "")"
25+
valid_drivers="docker-container|kubernetes|remote"
26+
if [[ "${build_driver}" =~ ^(${valid_drivers})$ ]]; then
27+
builder_instance_args+=(--driver "${build_driver}")
28+
else
29+
echo "+++ 🚨 Invalid driver: '${build_driver}'"
30+
echo "Valid Drivers: ${valid_drivers//|/, }"
31+
exit 1
32+
fi
33+
34+
if [[ "$(plugin_read_config BUILDER_BOOTSTRAP "true")" == "true" ]]; then
35+
builder_instance_args+=("--bootstrap")
36+
fi
37+
38+
if [[ "$(plugin_read_config BUILDER_DEBUG "false")" == "true" ]]; then
39+
builder_instance_args+=("--debug")
40+
fi
41+
42+
driver_opt="$(plugin_read_config BUILDER_DRIVER_OPT "")"
43+
if [[ -n "${driver_opt}" ]]; then
44+
builder_instance_args+=("--driver-opt ${driver_opt}")
45+
fi
46+
47+
builder_platform="$(plugin_read_config BUILDER_PLATFORM "")"
48+
if [[ -n "${builder_platform}" ]]; then
49+
builder_instance_args+=("--platform ${builder_platform}")
50+
fi
51+
52+
remote_address="$(plugin_read_config BUILDER_REMOTE_ADDRESS "")"
53+
if [[ -n "${remote_address}" ]]; then
54+
if [[ "${build_driver}" == "remote" ]]; then
55+
builder_instance_args+=("${remote_address}")
56+
else
57+
echo "+++ 🚨 Builder Driver '${build_driver}' used with 'remote-address' parameter"
58+
echo "'remote-address' can only be used with 'remote' driver"
59+
exit 1
60+
fi
61+
fi
62+
63+
echo "~~~ :docker: Creating Builder Instance '${builder_name}' with Driver '${build_driver}'"
64+
docker buildx create "${builder_instance_args[@]}"
65+
if [[ "${builder_use}" == "false" ]]; then
66+
echo "~~~ :warning: Builder Instance '${builder_name}' created but will not be used as 'use: true' parameter not specified"
67+
fi
68+
else
69+
echo "~~~ :docker: Not Creating Builder Instance '${builder_name}' as already exists"
70+
fi
71+
fi
72+
73+
if [[ "${builder_use}" == "true" ]]; then
74+
if builder_instance_exists "${builder_name}"; then
75+
echo "~~~ :docker: Using Builder Instance '${builder_name}'"
76+
docker buildx use "${builder_name}"
77+
else
78+
echo "+++ 🚨 Builder Instance '${builder_name}' does not exist"
79+
exit 1
80+
fi
81+
else
82+
current_builder=$(docker buildx inspect | grep -m 1 "Name" | awk '{print $2}') || true
83+
current_builder_driver=$(docker buildx inspect | grep -m 1 "Driver" | awk '{print $2}') || true
84+
echo "~~~ :docker: Using Default Builder '${current_builder}' with Driver '${current_builder_driver}'"
85+
fi

‎hooks/pre-exit

+26-1
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,35 @@ rm -f "docker-compose.buildkite-${BUILDKITE_BUILD_NUMBER}-override.yml"
1313

1414
# clean up resources after a run command. we do this here so that it will
1515
# run after a job is cancelled
16-
if [[ -n "$(plugin_read_list RUN)" ]] && [[ "$(plugin_read_config CLEANUP "true")" == "true" ]] ; then
16+
if [[ -n "$(plugin_read_list RUN)" ]] && [[ "$(plugin_read_config CLEANUP "true")" == "true" ]]; then
1717
# shellcheck source=lib/run.bash
1818
. "$DIR/../lib/run.bash"
1919

2020
echo "~~~ :docker: Cleaning up after docker-compose" >&2
2121
compose_cleanup
2222
fi
23+
24+
# clean up builder instances if specified
25+
builder_name="$(plugin_read_config BUILDER_NAME "")"
26+
builder_remove="$(plugin_read_config BUILDER_REMOVE "false")"
27+
28+
if [[ "${builder_remove}" == true ]]; then
29+
if builder_instance_exists "${builder_name}"; then
30+
31+
builder_remove_args=("${builder_name}")
32+
33+
if [[ "$(plugin_read_config BUILDER_KEEP_DAEMON "false")" == "true" ]]; then
34+
builder_remove_args+=("--keep-daemon")
35+
fi
36+
37+
if [[ "$(plugin_read_config BUILDER_KEEP_STATE "false")" == "true" ]]; then
38+
builder_remove_args+=("--keep-state")
39+
fi
40+
41+
echo "~~~ :docker: Cleaning up Builder Instance '${builder_name}'"
42+
docker buildx stop "${builder_name}"
43+
docker buildx rm "${builder_remove_args[@]}"
44+
else
45+
echo "~~~ :warning: Cannot remove Builder Instance '${builder_name}' as does not exist"
46+
fi
47+
fi

‎lib/shared.bash

+11
Original file line numberDiff line numberDiff line change
@@ -320,3 +320,14 @@ function validate_tag {
320320
return 1
321321
fi
322322
}
323+
324+
function builder_instance_exists() {
325+
local builder_name="$1"
326+
327+
# Check if the specified builder exists by suppressing output and checking the exit status
328+
if docker buildx inspect "${builder_name}" >/dev/null 2>&1; then
329+
return 0 # Builder exists
330+
else
331+
return 1 # Builder does not exist
332+
fi
333+
}

‎plugin.yml

+29
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,35 @@ configuration:
2323
build-alias:
2424
type: [ string, array ]
2525
minimum: 1
26+
builder:
27+
type: object
28+
properties:
29+
bootstrap:
30+
type: boolean
31+
create:
32+
type: boolean
33+
debug:
34+
type: boolean
35+
driver:
36+
type: string
37+
enum: [ "docker-container", "kubernetes", "remote" ]
38+
driver-opt:
39+
type: string
40+
keep-daemon:
41+
type: boolean
42+
keep-state:
43+
type: boolean
44+
name:
45+
type: string
46+
platform:
47+
type: string
48+
remote-address:
49+
type: string
50+
remove:
51+
type: boolean
52+
use:
53+
type: boolean
54+
additionalProperties: false
2655
build-labels:
2756
type: [ string, array ]
2857
minimum: 1

0 commit comments

Comments
 (0)