Skip to content

Commit ee8d1d9

Browse files
authored
Merge pull request #1855 from tlaurion/docker_helpers
add 3 Docker users/build helpers : local_dev, repro and latest, update README.md to simplify usage
2 parents 1939af6 + 4ec2fef commit ee8d1d9

File tree

4 files changed

+207
-12
lines changed

4 files changed

+207
-12
lines changed

README.md

Lines changed: 16 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -55,10 +55,10 @@ Build docker from nix develop layer locally
5555

5656
#### Build image
5757

58+
* Have docker and Nix installed
59+
5860
* Build nix developer local environment with flakes locked to specified versions
59-
* `nix --print-build-logs --verbose develop --ignore-environment --command true`
60-
* Build docker image with current develop created environment (this will take a while and create "linuxboot/heads:dev-env" local docker image):
61-
* `nix --print-build-logs --verbose build .#dockerImage && docker load < result`
61+
* `./docker_local_dev.sh`
6262

6363
On some hardened OSes, you may encounter problems with ptrace.
6464
```
@@ -75,12 +75,16 @@ sudo sysctl -w kernel.yama.ptrace_scope=1 #setup the value to let nix+docker run
7575

7676
Done!
7777

78-
Your local docker image "linuxboot/heads:dev-env" is ready to use, reproducible for the specific Heads commit used and will produce ROMs reproducible for that Heads commit ID.
78+
Your local docker image "linuxboot/heads:dev-env" is ready to use, reproducible for the specific Heads commit used to build it, and will produce ROMs reproducible for that Heads commit ID.
7979

8080
Jump into nix develop created docker image for interactive workflow
8181
====
82-
`docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) linuxboot/heads:dev-env`
82+
There is 3 helpers:
83+
- `./docker_local_dev.sh`: for developers wanting to customize docker image built from flake.nix(nix devenv creation) and flake.lock (pinned versions used by flake.nix)
84+
- `./docker_latest.sh`: for Heads developers, wanting to use latest published docker images to develop Heads
85+
- `./docker_repro.sh`: versioned docker image used under CircleCI to produce reproducivle builds, both locally and under CircleCI. **Use this one if in doubt**
8386

87+
ie: `./docker_repro.sh` will jump into CircleCI used versioned docker image for that Heads commit id to build images reproducibly if git repo is clean (not dirty).
8488

8589
From there you can use the docker image interactively.
8690

@@ -92,22 +96,22 @@ Please refer to [qemu documentation](targets/qemu.md) for more information.
9296

9397
Eg:
9498
```
95-
make BOARD=qemu-coreboot-fbwhiptail-tpm2 # Build rom, export public key to emulated usb storage from qemu runtime
96-
make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=~/pubkey.asc inject_gpg # Inject pubkey into rom image
97-
make BOARD=qemu-coreboot-fbwhiptail-tpm2 USB_TOKEN=Nitrokey3NFC PUBKEY_ASC=~/pubkey.asc ROOT_DISK_IMG=~/qemu-disks/debian-9.cow2 INSTALL_IMG=~/Downloads/debian-9.13.0-amd64-xfce-CD-1.iso run # Install
99+
./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 # Build rom, export public key to emulated usb storage from qemu runtime
100+
./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=~/pubkey.asc inject_gpg # Inject pubkey into rom image
101+
./docker_repro.sh make BOARD=qemu-coreboot-fbwhiptail-tpm2 USB_TOKEN=Nitrokey3NFC PUBKEY_ASC=~/pubkey.asc ROOT_DISK_IMG=~/qemu-disks/debian-9.cow2 INSTALL_IMG=~/Downloads/debian-9.13.0-amd64-xfce-CD-1.iso run # Install
98102
```
99103

100-
Alternatively, you can use locally built docker image to build a board ROM image in a single call.
104+
Alternatively, you can use locally built docker image to build a board ROM image in a single call **but do not expect reproducible builds if not using versioned docker images as per CircleCI as per usage of `./docker_repro.sh`**
101105

102106
Eg:
103-
`docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) linuxboot/heads:dev-env -- make BOARD=nitropad-nv41`
107+
`./docker_local_dev.sh make BOARD=nitropad-nv41`
104108

105109

106110
Pull docker hub image to prepare reproducible ROMs as CircleCI in one call
107111
====
108112
```
109-
docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=x230-hotp-maximized
110-
docker run -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- make BOARD=nitropad-nv41
113+
./docker_repro.sh make BOARD=x230-hotp-maximized
114+
./docker_repro.sh make BOARD=nitropad-nv41
111115
```
112116

113117
Maintenance notes on docker image

docker_latest.sh

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/bin/bash
2+
3+
# Inform the user that the latest published Docker image is being used
4+
echo "Using the latest Docker image: tlaurion/heads-dev-env:latest"
5+
6+
# Function to display usage information
7+
usage() {
8+
echo "Usage: $0 [OPTIONS] -- [COMMAND]"
9+
echo "Options:"
10+
echo " CPUS=N Set the number of CPUs"
11+
echo " V=1 Enable verbose mode"
12+
echo "Command:"
13+
echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME"
14+
}
15+
16+
# Function to kill GPG toolstack related processes using USB devices
17+
kill_usb_processes() {
18+
echo "Killing any GPG toolstack related processes on host currently using USB devices..."
19+
sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9
20+
if [ $? -ne 0 ]; then
21+
echo "Failed to kill GPG toolstack related processes using USB devices. Please run the following command manually:"
22+
echo "sudo lsof /dev/bus/usb/00*/0* | awk 'NR>1 {print \$2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print \$1}' | xargs -r sudo kill -9"
23+
exit 1
24+
fi
25+
}
26+
27+
# Handle Ctrl-C (SIGINT) to exit gracefully
28+
trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT
29+
30+
# Check if --help or -h is provided
31+
for arg in "$@"; do
32+
if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then
33+
usage
34+
exit 0
35+
fi
36+
done
37+
38+
# Kill processes using USB devices
39+
kill_usb_processes
40+
41+
# Inform the user about entering the Docker container
42+
echo "----"
43+
echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional."
44+
echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config."
45+
echo "----"
46+
echo "Entering the Docker container. Type 'exit' to return to the host shell."
47+
48+
# Execute the docker run command with the provided parameters
49+
docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) tlaurion/heads-dev-env:latest -- "$@"

docker_local_dev.sh

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
#!/bin/bash
2+
3+
# Check if Nix is installed
4+
if ! command -v nix &> /dev/null; then
5+
echo "Nix is not installed or not in the PATH. Please install Nix before running this script."
6+
echo "Refer to the README.md at the root of the repository for installation instructions."
7+
exit 1
8+
fi
9+
10+
# Check if Docker is installed
11+
if ! command -v docker &> /dev/null; then
12+
echo "Docker is not installed or not in the PATH. Please install Docker before running this script."
13+
echo "Refer to the README.md at the root of the repository for installation instructions."
14+
exit 1
15+
fi
16+
17+
# Inform the user about the Docker image being used
18+
echo "** This ./docker_local_dev.sh script is for developers usage only. **"
19+
echo ""
20+
echo "Using the last locally produced Docker image: linuxboot/heads:dev-env"
21+
echo "Warning: Using anything other than the published Docker image might lead to non-reproducible builds."
22+
echo ""
23+
echo "For using the latest published Docker image, refer to ./docker_latest.sh."
24+
echo "For producing reproducible builds as CircleCI, refer to ./docker_repro.sh."
25+
echo ""
26+
echo "---"
27+
28+
# Function to display usage information
29+
usage() {
30+
echo "Usage: $0 [OPTIONS] -- [COMMAND]"
31+
echo "Options:"
32+
echo " CPUS=N Set the number of CPUs"
33+
echo " V=1 Enable verbose mode"
34+
echo "Command:"
35+
echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME"
36+
}
37+
38+
# Function to kill GPG toolstack related processes using USB devices
39+
kill_usb_processes() {
40+
echo "Killing any GPG toolstack related processes on host currently using USB devices..."
41+
sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9
42+
if [ $? -ne 0 ]; then
43+
echo "Failed to kill GPG toolstack related processes using USB devices. Please run the following command manually:"
44+
echo "sudo lsof /dev/bus/usb/00*/0* | awk 'NR>1 {print \$2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print \$1}' | xargs -r sudo kill -9"
45+
exit 1
46+
fi
47+
}
48+
49+
# Handle Ctrl-C (SIGINT) to exit gracefully
50+
trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT
51+
52+
# Check if --help or -h is provided
53+
for arg in "$@"; do
54+
if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then
55+
usage
56+
exit 0
57+
fi
58+
done
59+
60+
# Check if the git repository is dirty and if flake.nix or flake.lock are part of the uncommitted changes
61+
if [ -n "$(git status --porcelain | grep -E 'flake\.nix|flake\.lock')" ]; then
62+
echo "Warning: Uncommitted changes detected in flake.nix or flake.lock. The Docker image will be rebuilt."
63+
echo "If this was not intended, please commit your changes and rerun the script."
64+
echo "Building the Docker image from flake.nix..."
65+
nix --print-build-logs --verbose develop --ignore-environment --command true
66+
nix --print-build-logs --verbose build .#dockerImage && docker load < result
67+
else
68+
echo "Git repository is clean. Using the previously built Docker image."
69+
echo "---"
70+
sleep 1
71+
fi
72+
73+
# Kill processes using USB devices
74+
kill_usb_processes
75+
76+
# Inform the user about entering the Docker container
77+
echo "----"
78+
echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional."
79+
echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config."
80+
echo "----"
81+
echo "Entering the Docker container. Type 'exit' to return to the host shell."
82+
83+
# Execute the docker run command with the provided parameters
84+
docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) linuxboot/heads:dev-env -- "$@"

docker_repro.sh

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
#!/bin/bash
2+
3+
# Extract the Docker image version from the CircleCI config file
4+
DOCKER_IMAGE=$(grep -oP '^\s*-?\s*image:\s*\K(tlaurion/heads-dev-env:[^\s]+)' .circleci/config.yml | head -n 1)
5+
6+
# Check if the Docker image was found
7+
if [ -z "$DOCKER_IMAGE" ]; then
8+
echo "Error: Docker image not found in .circleci/config.yml"
9+
exit 1
10+
fi
11+
12+
# Inform the user about the versioned CircleCI Docker image being used
13+
echo "Using CircleCI Docker image: $DOCKER_IMAGE"
14+
15+
# Function to display usage information
16+
usage() {
17+
echo "Usage: $0 [OPTIONS] -- [COMMAND]"
18+
echo "Options:"
19+
echo " CPUS=N Set the number of CPUs"
20+
echo " V=1 Enable verbose mode"
21+
echo "Command:"
22+
echo " The command to run inside the Docker container, e.g., make BOARD=BOARD_NAME"
23+
}
24+
25+
# Function to kill GPG toolstack related processes using USB devices
26+
kill_usb_processes() {
27+
echo "Killing any GPG toolstack related processes on host currently using USB devices..."
28+
sudo lsof /dev/bus/usb/00*/0* 2>/dev/null | awk 'NR>1 {print $2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print $1}' | xargs -r sudo kill -9
29+
if [ $? -ne 0 ]; then
30+
echo "Failed to kill GPG toolstack related processes using USB devices. Please run the following command manually:"
31+
echo "sudo lsof /dev/bus/usb/00*/0* | awk 'NR>1 {print \$2}' | xargs -r ps -p | grep -E 'scdaemon|pcscd' | awk '{print \$1}' | xargs -r sudo kill -9"
32+
exit 1
33+
fi
34+
}
35+
36+
# Handle Ctrl-C (SIGINT) to exit gracefully
37+
trap "echo 'Script interrupted. Exiting...'; exit 1" SIGINT
38+
39+
# Check if --help or -h is provided
40+
for arg in "$@"; do
41+
if [[ "$arg" == "--help" || "$arg" == "-h" ]]; then
42+
usage
43+
exit 0
44+
fi
45+
done
46+
47+
# Kill processes using USB devices
48+
kill_usb_processes
49+
50+
# Inform the user about entering the Docker container
51+
echo "----"
52+
echo "Usage reminder: The minimal command is 'make BOARD=XYZ', where additional options, including 'V=1' or 'CPUS=N' are optional."
53+
echo "For more advanced QEMU testing options, refer to targets/qemu.md and boards/qemu-*/*.config."
54+
echo "----"
55+
echo "Entering the Docker container. Type 'exit' to return to the host shell."
56+
57+
# Execute the docker run command with the provided parameters
58+
docker run --device=/dev/bus/usb:/dev/bus/usb -e DISPLAY=$DISPLAY --network host --rm -ti -v $(pwd):$(pwd) -w $(pwd) $DOCKER_IMAGE -- "$@"

0 commit comments

Comments
 (0)