Skip to content

Commit e2eb2f5

Browse files
oysandoysand
committed
Add script for building images locally
1 parent 94e07d3 commit e2eb2f5

2 files changed

Lines changed: 267 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -107,3 +107,40 @@ You may now run the tests with
107107
```bash
108108
uv run pytest -s .
109109
```
110+
111+
### Running against locally built images
112+
113+
By default the tests pull `ghcr.io/equinor/{flotilla-backend,sara,isar-robot}`. A change that
114+
spans armada *and* one of those services therefore cannot be verified until the service change
115+
is merged and an image published — even though the armada side is what proves the service side
116+
works.
117+
118+
To close that gap, build the images from your local working copies:
119+
120+
```bash
121+
scripts/build_local_images.sh # build and verify
122+
scripts/build_local_images.sh --run # ... and run the full suite against them
123+
```
124+
125+
The script expects the sibling checkouts of the superrepo (`../isar`, `../isar-robot`,
126+
`../flotilla`, `../sara`); override with `ISAR_DIR`, `ISAR_ROBOT_DIR`, `FLOTILLA_DIR`,
127+
`SARA_DIR`. It verifies each image before handing back, because a subtly broken build otherwise
128+
shows up only as an unexplained timeout several minutes into the suite.
129+
130+
The database schema is taken from the same local checkouts, via `MIGRATIONS_SOURCE_DIR` and
131+
`SARA_MIGRATIONS_SOURCE_DIR`, so application code and schema always agree. The directory is
132+
mounted read-only and copied into the migrations container, which means **uncommitted and
133+
untracked migrations are picked up**. Set either variable on its own if you want to mix a local
134+
schema with published images.
135+
136+
Two things worth knowing:
137+
138+
- **`flotilla`, `sara` and `isar` are built from the working tree**, so uncommitted changes are
139+
included. **`isar-robot` is cloned**, so only committed changes are — the script warns if that
140+
checkout is dirty. It has to be cloned because its Dockerfile bind-mounts `.git`, and in the
141+
superrepo that is a submodule *file* rather than a directory.
142+
- `isar-robot`'s `uv.lock` pins `isar` from PyPI, so the locally built `isar` wheel is installed
143+
over the released one.
144+
145+
The mosquitto broker is always the published image, and the OAuth2 mock is built automatically
146+
by the test fixtures.

‎scripts/build_local_images.sh‎

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Build the service images from local working copies and point the integration
4+
# tests at them, instead of the published :dev / :latest images.
5+
#
6+
# Why this exists
7+
# ---------------
8+
# The integration tests normally pull ghcr.io/equinor/{flotilla-backend,sara,
9+
# isar-robot}. That means a change which spans armada *and* one of the services
10+
# cannot be validated until the service change has been merged and an image
11+
# published -- but the armada side of the change is what proves the service side
12+
# works. This script closes that gap: build everything locally, run the suite,
13+
# then merge in confidence.
14+
#
15+
# Usage
16+
# -----
17+
# scripts/build_local_images.sh # build and verify the images
18+
# scripts/build_local_images.sh --run # ... and then run the full suite
19+
# scripts/build_local_images.sh --help
20+
#
21+
# Repository locations default to the superrepo sibling layout and can each be
22+
# overridden: ISAR_DIR ISAR_ROBOT_DIR FLOTILLA_DIR SARA_DIR
23+
24+
set -euo pipefail
25+
26+
TAG="${LOCAL_IMAGE_TAG:-local}"
27+
PLATFORM="linux/amd64"
28+
RUN_TESTS=false
29+
30+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
31+
ARMADA_DIR="$(dirname "$SCRIPT_DIR")"
32+
SIBLING_ROOT="$(dirname "$ARMADA_DIR")"
33+
34+
ISAR_DIR="${ISAR_DIR:-$SIBLING_ROOT/isar}"
35+
ISAR_ROBOT_DIR="${ISAR_ROBOT_DIR:-$SIBLING_ROOT/isar-robot}"
36+
FLOTILLA_DIR="${FLOTILLA_DIR:-$SIBLING_ROOT/flotilla}"
37+
SARA_DIR="${SARA_DIR:-$SIBLING_ROOT/sara}"
38+
39+
FLOTILLA_IMAGE="flotilla-backend:$TAG"
40+
SARA_IMAGE="sara:$TAG"
41+
ISAR_ROBOT_IMAGE="isar-robot:$TAG"
42+
ISAR_ROBOT_BASE_IMAGE="isar-robot:$TAG-base"
43+
44+
for arg in "$@"; do
45+
case "$arg" in
46+
--run) RUN_TESTS=true ;;
47+
--help|-h)
48+
# Print the header comment block, stopping at the first non-comment line.
49+
awk 'NR==1{next} /^#/{sub(/^# ?/,""); print; next} {exit}' "${BASH_SOURCE[0]}"
50+
exit 0 ;;
51+
*) echo "Unknown argument: $arg (try --help)" >&2; exit 2 ;;
52+
esac
53+
done
54+
55+
log() { printf '\n\033[1;34m==> %s\033[0m\n' "$*"; }
56+
warn() { printf '\033[1;33mWARNING: %s\033[0m\n' "$*" >&2; }
57+
die() { printf '\033[1;31mERROR: %s\033[0m\n' "$*" >&2; exit 1; }
58+
59+
require_dir() {
60+
[ -d "$1" ] || die "$2 not found at '$1'. Set $3 to override."
61+
}
62+
63+
require_dir "$ISAR_DIR" "isar repository" ISAR_DIR
64+
require_dir "$ISAR_ROBOT_DIR" "isar-robot repository" ISAR_ROBOT_DIR
65+
require_dir "$FLOTILLA_DIR" "flotilla repository" FLOTILLA_DIR
66+
require_dir "$SARA_DIR" "sara repository" SARA_DIR
67+
68+
docker info >/dev/null 2>&1 || die "Docker does not appear to be running."
69+
70+
TMP_DIR="$(mktemp -d)"
71+
cleanup() { rm -rf "$TMP_DIR"; }
72+
trap cleanup EXIT
73+
74+
# ---------------------------------------------------------------------------
75+
# flotilla-backend and sara are built straight from the working tree, so any
76+
# uncommitted changes are included.
77+
# ---------------------------------------------------------------------------
78+
79+
log "Building $FLOTILLA_IMAGE from $FLOTILLA_DIR"
80+
docker build --platform "$PLATFORM" \
81+
-f "$FLOTILLA_DIR/backend/Dockerfile" \
82+
-t "$FLOTILLA_IMAGE" \
83+
"$FLOTILLA_DIR/backend"
84+
85+
log "Building $SARA_IMAGE from $SARA_DIR"
86+
docker build --platform "$PLATFORM" -t "$SARA_IMAGE" "$SARA_DIR"
87+
88+
# ---------------------------------------------------------------------------
89+
# isar-robot needs two steps.
90+
#
91+
# 1. Its Dockerfile does `RUN --mount=source=.git,target=.git,type=bind`, and
92+
# setuptools_scm needs that git directory both to derive a version *and* to
93+
# discover package data such as src/isar_robot/example_data/. In the superrepo
94+
# the checkout is a submodule, so `.git` is a FILE ("gitdir: ...") which the
95+
# `!.git/` allowlist entry in .dockerignore does not match. Building directly
96+
# from the working tree therefore fails with "unable to detect version", and
97+
# forcing SETUPTOOLS_SCM_PRETEND_VERSION instead produces a wheel that is
98+
# missing example_data -- which only shows up much later as
99+
# RobotRetrieveInspectionException during a mission. Cloning into a temporary
100+
# directory yields a real .git directory with history and tags, so the stock
101+
# Dockerfile works unmodified.
102+
#
103+
# 2. isar-robot's uv.lock pins `isar` from PyPI (the lock is generated with
104+
# --no-sources, so the `[tool.uv.sources] isar = { path = "../isar" }` entry in
105+
# pyproject.toml is ignored). To test local isar changes, the locally built
106+
# wheel is installed over the released one.
107+
# ---------------------------------------------------------------------------
108+
109+
if [ -n "$(git -C "$ISAR_ROBOT_DIR" status --porcelain)" ]; then
110+
warn "$ISAR_ROBOT_DIR has uncommitted changes."
111+
warn "isar-robot is CLONED rather than built from the working tree, so those"
112+
warn "changes will NOT be in the image. Commit them first if they matter."
113+
fi
114+
115+
log "Cloning isar-robot into a temporary directory (needs a real .git)"
116+
git clone --quiet "$ISAR_ROBOT_DIR" "$TMP_DIR/isar-robot" \
117+
|| die "Failed to clone $ISAR_ROBOT_DIR"
118+
119+
log "Building $ISAR_ROBOT_BASE_IMAGE"
120+
docker build --platform "$PLATFORM" -t "$ISAR_ROBOT_BASE_IMAGE" "$TMP_DIR/isar-robot"
121+
122+
log "Building the isar wheel from $ISAR_DIR (working tree, uncommitted changes included)"
123+
mkdir -p "$TMP_DIR/wheels"
124+
if ! ( cd "$ISAR_DIR" && uv build --wheel -o "$TMP_DIR/wheels" ) >"$TMP_DIR/uv-build.log" 2>&1; then
125+
cat "$TMP_DIR/uv-build.log" >&2
126+
die "Failed to build the isar wheel"
127+
fi
128+
ISAR_WHEEL="$(ls "$TMP_DIR"/wheels/isar-*.whl 2>/dev/null | head -1)"
129+
[ -n "$ISAR_WHEEL" ] || die "No isar wheel was produced in $TMP_DIR/wheels"
130+
echo "Built $(basename "$ISAR_WHEEL")"
131+
132+
log "Overlaying the local isar onto $ISAR_ROBOT_IMAGE"
133+
# The wheel keeps its original filename: uv rejects anything that is not a valid
134+
# PEP 427 wheel name ("Must have a Python tag").
135+
mkdir -p "$TMP_DIR/overlay/wheels"
136+
cp "$ISAR_WHEEL" "$TMP_DIR/overlay/wheels/"
137+
cat > "$TMP_DIR/overlay/Dockerfile" <<OVERLAY
138+
FROM $ISAR_ROBOT_BASE_IMAGE
139+
USER root
140+
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
141+
COPY wheels /tmp/wheels
142+
# --no-deps keeps the resolved dependency set from the base image. If isar ever
143+
# gains a new dependency this will need revisiting.
144+
RUN /bin/uv pip install --python /app/.venv/bin/python --no-deps --reinstall /tmp/wheels/*.whl \
145+
&& rm -rf /tmp/wheels
146+
USER 1000
147+
CMD ["isar-start"]
148+
OVERLAY
149+
docker build --platform "$PLATFORM" -t "$ISAR_ROBOT_IMAGE" "$TMP_DIR/overlay"
150+
151+
# ---------------------------------------------------------------------------
152+
# Verify. These checks take seconds; without them a bad image only reveals
153+
# itself several minutes into the test suite, as a timeout with no obvious cause.
154+
# ---------------------------------------------------------------------------
155+
156+
log "Verifying images"
157+
158+
verify_dotnet_settings() {
159+
local image="$1"
160+
if docker run --rm --platform "$PLATFORM" --entrypoint sh "$image" \
161+
-c 'test -f /app/appsettings.IntegrationTest.json'; then
162+
echo " OK $image has appsettings.IntegrationTest.json"
163+
else
164+
die "$image is missing /app/appsettings.IntegrationTest.json"
165+
fi
166+
}
167+
168+
verify_dotnet_settings "$FLOTILLA_IMAGE"
169+
verify_dotnet_settings "$SARA_IMAGE"
170+
171+
docker run --rm --platform "$PLATFORM" \
172+
--entrypoint /app/.venv/bin/python "$ISAR_ROBOT_IMAGE" -c '
173+
import pathlib, sys
174+
import isar_robot
175+
from isar.config.settings import settings
176+
177+
problems = []
178+
179+
example_data = pathlib.Path(isar_robot.__file__).parent / "example_data"
180+
count = len(list(example_data.iterdir())) if example_data.is_dir() else 0
181+
if count == 0:
182+
problems.append(
183+
"isar_robot/example_data is missing or empty; the wheel was built without a "
184+
"usable git directory, and missions will fail with "
185+
"RobotRetrieveInspectionException"
186+
)
187+
188+
if "OPENID_CONFIG_URL" not in type(settings).model_fields:
189+
problems.append(
190+
"isar does not expose OPENID_CONFIG_URL; the local isar overlay did not take"
191+
)
192+
193+
for problem in problems:
194+
print(" FAIL " + problem)
195+
if problems:
196+
sys.exit(1)
197+
198+
print(f" OK isar-robot has {count} example_data files")
199+
print(" OK isar exposes OPENID_CONFIG_URL")
200+
' || die "isar-robot image verification failed"
201+
202+
# ---------------------------------------------------------------------------
203+
204+
PYTEST_ENV=(
205+
"FLOTILLA_BACKEND_IMAGE=$FLOTILLA_IMAGE"
206+
"SARA_IMAGE=$SARA_IMAGE"
207+
"ISAR_ROBOT_IMAGE=$ISAR_ROBOT_IMAGE"
208+
# Take the database schema from the same checkouts the images were built
209+
# from, rather than cloning the app repositories from GitHub. Without this
210+
# you would run local application code against a remote schema, and any
211+
# migration that is unpushed or uncommitted would be missed entirely.
212+
"MIGRATIONS_SOURCE_DIR=$FLOTILLA_DIR"
213+
"SARA_MIGRATIONS_SOURCE_DIR=$SARA_DIR"
214+
)
215+
216+
log "Images ready"
217+
printf ' %s\n' "$FLOTILLA_IMAGE" "$SARA_IMAGE" "$ISAR_ROBOT_IMAGE"
218+
219+
if [ "$RUN_TESTS" = true ]; then
220+
log "Running the integration tests against the local images"
221+
cd "$ARMADA_DIR"
222+
env "${PYTEST_ENV[@]}" uv run --frozen pytest -n auto robotics_integration_tests
223+
else
224+
log "Run the integration tests with:"
225+
echo
226+
printf ' cd %s\n' "$ARMADA_DIR"
227+
for pair in "${PYTEST_ENV[@]}"; do printf ' %s \\\n' "$pair"; done
228+
printf ' uv run --frozen pytest -n auto robotics_integration_tests\n\n'
229+
printf 'Or re-run this script with --run.\n\n'
230+
fi

0 commit comments

Comments
 (0)