fix(tee-vm): pin a test-scoped metrics port in the SNP E2E harness #122
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: amdsev-initrd-test | |
| # Builds the canonical sealed TEE-VM initrd (same invocation as the | |
| # amdsev-release workflow, minus OVMF) whenever misc/AMDSEV changes, then | |
| # boot-tests it with misc/AMDSEV/scripts/test-initrd.sh: plain-QEMU direct | |
| # kernel boot, guest init comes up, Katana is started through the | |
| # virtio-serial control channel, the forwarded RPC endpoint answers | |
| # starknet_chainId, and the graceful `stop` command powers the guest off. | |
| # | |
| # The embedded katana is built FROM THIS REF via scripts/build-gnu.sh | |
| # --native — the profile and feature set of the published `_native` | |
| # linux_amd64 binary (the one amdsev-release.yml embeds) — so the boot test | |
| # validates the tree under test end to end (a PR that breaks `--tee sev-snp` | |
| # or the in-guest cairo-native link path fails here, before any release). | |
| # | |
| # Runner | |
| # ------ | |
| # Stock ubuntu-latest. No SEV-SNP hardware is needed: the smoke test boots | |
| # without OVMF/SEV, and the sealed-mode initrd takes its unsealed mount path | |
| # (plain ext4 on /dev/sda) when the cmdline carries no | |
| # KATANA_EXPECTED_LUKS_UUID. Docker (for the static cryptsetup + ld builds) | |
| # and /dev/kvm are both available on ubuntu-latest. | |
| on: | |
| push: | |
| branches: [main] | |
| paths: | |
| - "misc/AMDSEV/**" | |
| - ".github/workflows/amdsev-initrd-test.yml" | |
| pull_request: | |
| paths: | |
| - "misc/AMDSEV/**" | |
| - ".github/workflows/amdsev-initrd-test.yml" | |
| concurrency: | |
| group: amdsev-initrd-test-${{ github.ref }} | |
| cancel-in-progress: true | |
| defaults: | |
| run: | |
| working-directory: misc/AMDSEV | |
| jobs: | |
| build-katana: | |
| runs-on: ubuntu-latest-32-cores | |
| # 60 (was 45): the native feature statically links LLVM/MLIR and fat-LTOs | |
| # the result — cold-cache builds run well past the old budget. | |
| timeout-minutes: 60 | |
| # katana-dev carries the full build toolchain including asdf/scarb for | |
| # the contract artifacts that katana-contracts embeds at compile time. | |
| container: | |
| image: ghcr.io/dojoengine/katana-dev:latest | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| submodules: recursive | |
| # Workaround for https://github.com/actions/runner-images/issues/6775 | |
| - run: git config --global --add safe.directory "*" | |
| working-directory: . | |
| - uses: Swatinem/rust-cache@v2 | |
| with: | |
| # -native suffix: the native feature set compiles a very different | |
| # dep graph (cairo-native, melior, llvm-sys) — don't thrash caches | |
| # from the old portable builds. | |
| key: amdsev-build-katana-native | |
| - name: Restore cached contract artifacts | |
| id: contracts-cache | |
| uses: actions/cache@v4 | |
| with: | |
| path: crates/contracts/build | |
| # Same hash inputs as test.yml's fixtures cache (a broad | |
| # crates/contracts/** glob makes hashFiles fail on this tree). | |
| key: amdsev-contracts-${{ hashFiles('Makefile', 'crates/contracts/build.rs', 'crates/contracts/contracts/account/**', 'crates/contracts/contracts/legacy/**', 'crates/contracts/contracts/messaging/**', 'crates/contracts/contracts/test-contracts/**', 'crates/contracts/contracts/vrf/**', 'crates/contracts/contracts/avnu/**', 'crates/contracts/contracts/openzeppelin/**', 'crates/contracts/contracts/piltover/**', 'crates/contracts/contracts/Scarb.toml') }} | |
| - name: Generate contract artifacts | |
| if: steps.contracts-cache.outputs.cache-hit != 'true' | |
| working-directory: . | |
| run: make contracts | |
| - name: Build katana (native release configuration) | |
| # build-gnu.sh --native matches the FEATURE SET of the published | |
| # `_native` linux_amd64 binary (the asset amdsev-release.yml embeds): | |
| # performance profile plus client,init-slot,jemalloc,tee-snp, | |
| # tee-mock,native. The katana-dev container ships LLVM/MLIR 19 with | |
| # MLIR_SYS_190_PREFIX etc. preset, so no extra install step. The | |
| # container is noble-based (glibc 2.39), matching the initrd's pinned | |
| # GLIBC_RUNTIME_PACKAGES, so the binary's symbol requirements cannot | |
| # exceed what the guest ships. | |
| working-directory: . | |
| run: ./scripts/build-gnu.sh --native | |
| - name: Upload katana binary | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: amdsev-katana-binary | |
| retention-days: 1 | |
| if-no-files-found: error | |
| path: target/x86_64-unknown-linux-gnu/performance/katana | |
| build-and-boot: | |
| needs: build-katana | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - name: Pin SOURCE_DATE_EPOCH to HEAD commit time | |
| # Same rationale as amdsev-release.yml: without this, build.sh falls | |
| # back to wall clock and the initrd bytes differ on every run of the | |
| # same source. Not strictly required for a smoke test, but it keeps | |
| # CI builds byte-comparable to local builds of the same ref. | |
| run: echo "SOURCE_DATE_EPOCH=$(git log -1 --format=%ct)" >> "$GITHUB_ENV" | |
| - name: Install build and test dependencies | |
| run: | | |
| sudo apt-get update | |
| # musl-tools: linker for the snp-derivekey musl build | |
| # zstd, cpio: initrd packaging | |
| # qemu-system-x86: plain-QEMU boot smoke test (no SEV needed) | |
| # socat: control-channel client used by test-initrd.sh | |
| sudo apt-get install -y --no-install-recommends \ | |
| musl-tools zstd cpio qemu-system-x86 socat | |
| - name: Install Rust musl target | |
| # Run inside snp-tools so its rust-toolchain.toml selects the same | |
| # toolchain build.sh will use for the snp-derivekey build. | |
| working-directory: misc/AMDSEV/snp-tools | |
| run: rustup target add x86_64-unknown-linux-musl | |
| - name: Enable KVM for the runner user | |
| # ubuntu-latest exposes /dev/kvm but root-only by default. | |
| # test-initrd.sh falls back to TCG emulation without it, which makes | |
| # the Katana boot far slower and flakier under the test timeout. | |
| run: | | |
| echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \ | |
| | sudo tee /etc/udev/rules.d/99-kvm4all.rules | |
| sudo udevadm control --reload-rules | |
| sudo udevadm trigger --name-match=kvm | |
| - name: Download katana binary built from this ref | |
| uses: actions/download-artifact@v5 | |
| with: | |
| name: amdsev-katana-binary | |
| path: misc/AMDSEV/.katana-bin | |
| - name: Stage katana binary | |
| # Artifact transport drops the execute bit. | |
| run: | | |
| chmod +x .katana-bin/katana | |
| echo "KATANA_BIN=$PWD/.katana-bin/katana" >> "$GITHUB_ENV" | |
| - name: Download sidecar binaries (paymaster-service, vrf-server) | |
| # The PR ref has no matching katana release, so pull the sidecars | |
| # from the latest published katana release — this test exercises the | |
| # bundling path (preflight, multi-root dynamic-lib walk incl. the | |
| # libssl pin, cpio modes, reproducibility), not sidecar behavior, so | |
| # exact sidecar bytes don't matter here. | |
| env: | |
| GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} | |
| run: | | |
| KATANA_VER="$(gh release list --repo dojoengine/katana --exclude-drafts --limit 50 \ | |
| --json tagName --jq '[.[].tagName | select(test("^v[0-9]"))][0]')" | |
| [ -n "$KATANA_VER" ] || { echo "could not resolve latest katana release" >&2; exit 1; } | |
| echo "Using sidecars from katana release: $KATANA_VER" | |
| mkdir -p .sidecar-dl | |
| gh release download "$KATANA_VER" \ | |
| --repo dojoengine/katana \ | |
| --pattern 'paymaster-service_*_linux_amd64.tar.gz' \ | |
| --pattern 'vrf-server_*_linux_amd64.tar.gz' \ | |
| --dir .sidecar-dl/ | |
| tar -xzf .sidecar-dl/paymaster-service_*_linux_amd64.tar.gz -C .sidecar-dl/ | |
| tar -xzf .sidecar-dl/vrf-server_*_linux_amd64.tar.gz -C .sidecar-dl/ | |
| PAYMASTER_BIN="$(find "$PWD/.sidecar-dl" -type f -name paymaster-service | head -n1)" | |
| VRF_BIN="$(find "$PWD/.sidecar-dl" -type f -name vrf-server | head -n1)" | |
| if [ -z "$PAYMASTER_BIN" ] || [ -z "$VRF_BIN" ]; then | |
| echo "sidecar binary missing after extract" >&2 | |
| exit 1 | |
| fi | |
| chmod +x "$PAYMASTER_BIN" "$VRF_BIN" | |
| { | |
| echo "PAYMASTER_BIN=$PAYMASTER_BIN" | |
| echo "VRF_BIN=$VRF_BIN" | |
| } >> "$GITHUB_ENV" | |
| - name: Build kernel + sealed initrd | |
| # Skips OVMF — the EDK2 build takes far longer than everything else | |
| # combined and the plain-QEMU boot test doesn't load firmware. The | |
| # kernel component just downloads the pinned Ubuntu .deb (vmlinuz is | |
| # needed for direct kernel boot below). | |
| run: | | |
| ./build.sh --katana "$KATANA_BIN" \ | |
| --paymaster-bin "$PAYMASTER_BIN" --vrf-bin "$VRF_BIN" \ | |
| kernel initrd | |
| - name: Assert sidecars and native-linking inputs are bundled in the initrd | |
| run: | | |
| zcat output/qemu/initrd.img | cpio -t 2>/dev/null > /tmp/initrd-list | |
| rc=0 | |
| # bin/ld + lib64/libc.so + libc_nonshared.a are the cairo-native | |
| # runtime-link inputs (see build-initrd.sh's "cairo-native link | |
| # inputs" section). | |
| for f in bin/paymaster-service bin/vrf-server usr/lib/ssl/cert.pem \ | |
| bin/ld lib64/libc.so usr/lib/x86_64-linux-gnu/libc_nonshared.a; do | |
| grep -q "^\.\?/\?$f\$" /tmp/initrd-list || { echo "::error::$f missing from initrd"; rc=1; } | |
| done | |
| grep -q 'libssl\.so\.3' /tmp/initrd-list || { echo "::error::libssl.so.3 missing from initrd"; rc=1; } | |
| grep -q 'libcrypto\.so\.3' /tmp/initrd-list || { echo "::error::libcrypto.so.3 missing from initrd"; rc=1; } | |
| exit "$rc" | |
| - name: Reproducibility check — rebuild initrd, compare digests | |
| # Byte-identical initrds are the core promise: the SEV-SNP launch | |
| # measurement hashes the initrd, so verifiers can only reproduce a | |
| # published measurement if the build is deterministic. Same | |
| # SOURCE_DATE_EPOCH, same pins, fresh output dir — any digest drift | |
| # here means nondeterminism crept into the packaging (unsorted | |
| # globs, leaked timestamps, host paths). The second build reuses the | |
| # already-built static cryptsetup/snp-derivekey/ld (skip-if-cached), | |
| # so it only repeats the deb downloads and cpio packaging. | |
| run: | | |
| first="$(sha256sum output/qemu/initrd.img | awk '{print $1}')" | |
| ./build.sh --katana "$KATANA_BIN" \ | |
| --paymaster-bin "$PAYMASTER_BIN" --vrf-bin "$VRF_BIN" \ | |
| --install output/repro initrd | |
| second="$(sha256sum output/repro/initrd.img | awk '{print $1}')" | |
| echo "first build: $first" | |
| echo "second build: $second" | |
| if [ "$first" != "$second" ]; then | |
| echo "::error::initrd build is NOT reproducible — digests differ across identical builds" | |
| exit 1 | |
| fi | |
| echo "Reproducible: digests match" | |
| - name: Boot smoke test | |
| # --output-dir is required: the script's default resolves relative to | |
| # scripts/, not the directory where build.sh writes its artifacts. | |
| # KATANA_TEST_NATIVE=1 boots with --enable-native-compilation and | |
| # exercises the in-guest cairo-native link path (bundled /bin/ld + | |
| # generated /lib64/libc.so) — valid because the katana under test is | |
| # a --native build. | |
| run: KATANA_TEST_NATIVE=1 ./scripts/test-initrd.sh --output-dir output/qemu --timeout 300 | |
| - name: Upload initrd for debugging | |
| # Only on failure — the artifact exists to reproduce a failed boot | |
| # locally (qemu -kernel/-initrd against the exact CI-built image). | |
| if: failure() | |
| uses: actions/upload-artifact@v4 | |
| with: | |
| name: initrd-debug | |
| path: | | |
| misc/AMDSEV/output/qemu/initrd.img | |
| misc/AMDSEV/output/qemu/build-info.txt | |
| misc/AMDSEV/output/repro/initrd.img | |
| if-no-files-found: ignore |