Skip to content

Commit 8ca350c

Browse files
committed
feat(test-guest): add snap lifecycle reproduction harness
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent 72b9c4a commit 8ca350c

5 files changed

Lines changed: 242 additions & 3 deletions

File tree

nix/test-guest/README.md

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,11 @@ The root [`flake.nix`](../../flake.nix) exposes this directory as the `test-gues
5757
| Fedora 44 | No | Yes | Yes | `.rpm` |
5858
| Rocky Linux 9 | Yes | Yes | Yes | `.rpm` |
5959

60+
The `snapd` configuration is available for Ubuntu and prepares snapd for
61+
local Snap lifecycle experiments. It does not install Docker, because the Snap
62+
gateway reproduction uses the Docker **Snap** and its `docker:docker-daemon`
63+
interface rather than the host-package Docker configuration.
64+
6065
The Ubuntu 24.04 Podman configuration is available for runtime and packaging
6166
checks, but its Podman 4 release does not provide the `pasta` rootless network
6267
helper required by OpenShell sandbox callbacks. OpenShell Podman E2E runs use
@@ -117,7 +122,9 @@ Configurations are Ansible playbooks stored under `nix/test-guest/configuration/
117122

118123
Configurations run in the order provided on the command line. OpenShell packages and copied binaries are installed after all configurations succeed.
119124

120-
`--install` packages and `--copy` executables are applied by a dedicated per-run Ansible playbook. They are not stored in prepared VM cache entries.
125+
`--install` packages, `--copy` executables, and `--copy-file` regular files are
126+
applied by a dedicated per-run Ansible playbook. They are not stored in prepared
127+
VM cache entries.
121128

122129
## Prepared VM cache
123130

@@ -217,6 +224,29 @@ nix run .#test-guest -- \
217224
-- openshell --version
218225
```
219226

227+
## Reproduce Snap gateway startup
228+
229+
The gateway Snap must be native to the guest architecture. Copy an existing
230+
Snap artifact and the reproduction script into a prepared Ubuntu guest, then
231+
run the script as root. It follows the Release Canary ordering exactly: install
232+
the Snap, connect Docker/log/system interfaces, and immediately query the
233+
gateway. On each failure it prints snapd and gateway journals.
234+
235+
```shell
236+
nix run .#test-guest -- \
237+
--distro ubuntu \
238+
--with snapd \
239+
--keep \
240+
--copy-file ./openshell_*.snap:/tmp/openshell.snap \
241+
--copy ./nix/test-guest/scripts/snap-gateway-repro.sh:/usr/local/bin/snap-gateway-repro \
242+
-- sudo /usr/local/bin/snap-gateway-repro /tmp/openshell.snap 10 30
243+
```
244+
245+
`--keep` retains the overlay and serial log when diagnosing a failure. The
246+
runner prints their location after shutdown. The final `30` accepts automatic
247+
recovery for up to 30 seconds; omit it to require the canary's immediate check.
248+
249+
220250
The destination must be an absolute guest path. Copied files are installed with mode `0755`.
221251

222252
## Runner options
@@ -226,6 +256,8 @@ The destination must be an absolute guest path. Copied files are installed with
226256
--with NAME Apply docker, podman, or selinux; repeatable
227257
--install PATH Install a .deb or .rpm package; repeatable
228258
--copy SRC:DEST Copy an executable into the guest; repeatable
259+
--copy-file SRC:DEST
260+
Copy a regular file into the guest; repeatable
229261
--ssh-port PORT Use a specific loopback SSH forwarding port
230262
--forward-port HOST_PORT:GUEST_PORT
231263
Forward a loopback host port to a guest port; repeatable
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
- name: Configure snapd
6+
hosts: test_vm
7+
become: true
8+
gather_facts: true
9+
10+
tasks:
11+
- name: Validate snapd support
12+
ansible.builtin.assert:
13+
that:
14+
- ansible_facts.distribution == "Ubuntu"
15+
fail_msg: >-
16+
snapd is currently configured only for Ubuntu test guests,
17+
not {{ ansible_facts.distribution }}.
18+
19+
- name: Refresh Ubuntu package metadata
20+
ansible.builtin.apt:
21+
update_cache: true
22+
23+
- name: Install snapd
24+
ansible.builtin.apt:
25+
name: snapd
26+
state: present
27+
install_recommends: false
28+
29+
- name: Start snapd socket activation
30+
ansible.builtin.systemd_service:
31+
name: snapd.socket
32+
enabled: true
33+
state: started
34+
35+
- name: Wait for snapd seed
36+
ansible.builtin.command:
37+
cmd: snap wait system seed.loaded
38+
changed_when: false

nix/test-guest/default.nix

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ let
2424
docker = ./configuration/docker.yml;
2525
podman = ./configuration/podman.yml;
2626
selinux = ./configuration/selinux.yml;
27+
snapd = ./configuration/snapd.yml;
2728
};
2829

2930
mkDistroProfile =

nix/test-guest/run.sh

Lines changed: 55 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,11 @@ Usage:
1313
1414
Options:
1515
--distro NAME Base distro: ubuntu, centos, fedora, or rocky
16-
--with NAME Apply a configuration; repeatable (docker, podman, selinux)
16+
--with NAME Apply a configuration; repeatable (docker, podman, selinux, snapd)
1717
--install PATH Install a .deb or .rpm package; repeatable
1818
--copy SRC:DEST Copy an executable to an absolute guest path; repeatable
19+
--copy-file SRC:DEST
20+
Copy a regular file to an absolute guest path; repeatable
1921
--ssh-port PORT Use a specific loopback SSH forwarding port
2022
--forward-port HOST_PORT:GUEST_PORT
2123
Forward a loopback host port to a guest port; repeatable
@@ -50,6 +52,7 @@ list=0
5052
configurations=()
5153
packages=()
5254
copies=()
55+
files=()
5356
forward_ports=()
5457
guest_command=()
5558

@@ -75,6 +78,11 @@ while [ "$#" -gt 0 ]; do
7578
copies+=("$2")
7679
shift 2
7780
;;
81+
--copy-file)
82+
require_value "$@"
83+
files+=("$2")
84+
shift 2
85+
;;
7886
--ssh-port)
7987
require_value "$@"
8088
requested_ssh_port=$2
@@ -251,6 +259,36 @@ for copy_spec in "${copies[@]}"; do
251259
done
252260
copies=("${resolved_copies[@]}")
253261

262+
resolved_files=()
263+
for file_spec in "${files[@]}"; do
264+
source_path=${file_spec%%:*}
265+
destination=${file_spec#*:}
266+
if [ "${source_path}" = "${file_spec}" ] ||
267+
! source_path=$(realpath -- "${source_path}") ||
268+
[ ! -f "${source_path}" ]; then
269+
echo "invalid --copy-file source: ${file_spec}" >&2
270+
exit 2
271+
fi
272+
case "${destination}" in
273+
/*)
274+
if [[ ${destination} == *"/../"* ]] || [[ ${destination} == */.. ]]; then
275+
echo "--copy-file destination must not contain '..': ${destination}" >&2
276+
exit 2
277+
fi
278+
if [[ ! ${destination} =~ ^/[A-Za-z0-9._+~/-]+$ ]]; then
279+
echo "--copy-file destination contains unsupported characters: ${destination}" >&2
280+
exit 2
281+
fi
282+
;;
283+
*)
284+
echo "--copy-file destination must be absolute: ${destination}" >&2
285+
exit 2
286+
;;
287+
esac
288+
resolved_files+=("${source_path}:${destination}")
289+
done
290+
files=("${resolved_files[@]}")
291+
254292
test_vm_cpu=host
255293
ssh_wait_seconds=180
256294
if [ "${TEST_GUEST_ACCELERATOR}" = kvm ] &&
@@ -583,7 +621,7 @@ else
583621
echo "==> Reusing cached configuration: ${configurations[*]:-base image}"
584622
fi
585623

586-
if [ "${#packages[@]}" -gt 0 ] || [ "${#copies[@]}" -gt 0 ]; then
624+
if [ "${#packages[@]}" -gt 0 ] || [ "${#copies[@]}" -gt 0 ] || [ "${#files[@]}" -gt 0 ]; then
587625
phase_started_at=${SECONDS}
588626
artifact_staging_dir=/tmp/openshell-test-guest-artifacts-$$
589627
ssh "${ssh_args[@]}" openshell@127.0.0.1 \
@@ -629,6 +667,21 @@ if [ "${#packages[@]}" -gt 0 ] || [ "${#copies[@]}" -gt 0 ]; then
629667
artifact_index=$((artifact_index + 1))
630668
done
631669

670+
artifact_index=0
671+
for file_spec in "${files[@]}"; do
672+
source_path=${file_spec%%:*}
673+
destination=${file_spec#*:}
674+
remote_path=${artifact_staging_dir}/file-${artifact_index}
675+
echo "==> Copying file: ${destination}"
676+
scp -q "${scp_args[@]}" \
677+
"${source_path}" "openshell@127.0.0.1:${remote_path}"
678+
printf -v install_command \
679+
'sudo install -D -m 0644 -- %q %q' \
680+
"${remote_path}" "${destination}"
681+
ssh "${ssh_args[@]}" openshell@127.0.0.1 "${install_command}"
682+
artifact_index=$((artifact_index + 1))
683+
done
684+
632685
ssh "${ssh_args[@]}" openshell@127.0.0.1 \
633686
"rm -rf -- '${artifact_staging_dir}'"
634687
report_timing "artifact transfer" "${phase_started_at}"
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
#!/usr/bin/env bash
2+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
# Reproduce the Release Canary Snap lifecycle: install the OpenShell Snap,
6+
# connect its interfaces after the daemon is started, then immediately use the
7+
# local gateway. Run this as root inside an Ubuntu guest prepared with --with snapd.
8+
9+
set -uo pipefail
10+
11+
usage() {
12+
cat <<'EOF'
13+
Usage: snap-gateway-repro.sh SNAP_FILE [ATTEMPTS] [READY_TIMEOUT_SECONDS]
14+
15+
Install SNAP_FILE repeatedly using the Release Canary interface ordering.
16+
ATTEMPTS defaults to 1. READY_TIMEOUT_SECONDS defaults to 0, preserving the
17+
canary's immediate readiness check. Set it to a positive value to wait for
18+
automatic gateway recovery after the immediate check fails. Every failed
19+
attempt prints service, connection, snap-change, journal, gateway-log, and
20+
listener diagnostics.
21+
EOF
22+
}
23+
24+
if [ "$#" -lt 1 ] || [ "$#" -gt 3 ]; then
25+
usage >&2
26+
exit 2
27+
fi
28+
29+
snap_file=$1
30+
attempts=${2:-1}
31+
ready_timeout=${3:-0}
32+
if [ ! -f "${snap_file}" ]; then
33+
echo "Snap file does not exist: ${snap_file}" >&2
34+
exit 2
35+
fi
36+
if [[ ! ${attempts} =~ ^[1-9][0-9]*$ ]]; then
37+
echo "ATTEMPTS must be a positive integer: ${attempts}" >&2
38+
exit 2
39+
fi
40+
if [[ ! ${ready_timeout} =~ ^[0-9]+$ ]]; then
41+
echo "READY_TIMEOUT_SECONDS must be a non-negative integer: ${ready_timeout}" >&2
42+
exit 2
43+
fi
44+
45+
diagnostics() {
46+
local attempt=$1
47+
echo "========== Snap diagnostics (attempt ${attempt}) ==========" >&2
48+
snap services openshell >&2 || true
49+
snap connections openshell >&2 || true
50+
snap changes >&2 || true
51+
systemctl status snap.openshell.gateway.service --no-pager >&2 || true
52+
journalctl -b -u snap.openshell.gateway.service --no-pager -n 300 >&2 || true
53+
journalctl -b -u snapd.service --no-pager -n 300 >&2 || true
54+
snap logs openshell.gateway -n=300 >&2 || true
55+
ss -ltnp '( sport = :17670 )' >&2 || true
56+
}
57+
58+
gateway_is_ready() {
59+
runuser -u openshell -- /snap/bin/openshell status >/dev/null 2>&1
60+
}
61+
62+
wait_for_gateway() {
63+
local deadline=$((SECONDS + ready_timeout))
64+
while [ "${SECONDS}" -lt "${deadline}" ]; do
65+
if gateway_is_ready; then
66+
return 0
67+
fi
68+
sleep 1
69+
done
70+
gateway_is_ready
71+
}
72+
73+
if ! snap list docker >/dev/null 2>&1; then
74+
echo "==> Installing Docker Snap"
75+
snap install docker
76+
fi
77+
78+
failures=0
79+
for attempt in $(seq 1 "${attempts}"); do
80+
echo "==> Snap gateway reproduction attempt ${attempt}/${attempts}"
81+
snap remove --purge openshell >/dev/null 2>&1 || true
82+
rm -rf /home/openshell/snap/openshell
83+
84+
if ! snap install "${snap_file}" --dangerous ||
85+
! snap connect openshell:docker docker:docker-daemon ||
86+
! snap connect openshell:log-observe ||
87+
! snap connect openshell:system-observe; then
88+
echo "OpenShell installation or interface connection failed" >&2
89+
diagnostics "${attempt}"
90+
failures=$((failures + 1))
91+
continue
92+
fi
93+
94+
# This deliberately does not wait for the listener. It mirrors the canary
95+
# and exposes a daemon that fails or races after late interface connections.
96+
if ! runuser -u openshell -- /snap/bin/openshell gateway add \
97+
http://127.0.0.1:17670 --local --name snap-docker ||
98+
! runuser -u openshell -- /snap/bin/openshell gateway select snap-docker ||
99+
! gateway_is_ready; then
100+
if [ "${ready_timeout}" -gt 0 ] && wait_for_gateway; then
101+
echo "Gateway recovered automatically within ${ready_timeout}s"
102+
continue
103+
fi
104+
echo "Gateway was not usable immediately after interface connection" >&2
105+
diagnostics "${attempt}"
106+
failures=$((failures + 1))
107+
fi
108+
done
109+
110+
if [ "${failures}" -gt 0 ]; then
111+
echo "${failures}/${attempts} attempt(s) failed" >&2
112+
exit 1
113+
fi
114+
115+
echo "All ${attempts} attempt(s) passed"

0 commit comments

Comments
 (0)