Skip to content

Commit 1594816

Browse files
kvinwangkvinwang
committed
ci(sdk): run released 0.5.x SDK suites against the current agent
dstack 0.6.0 froze the unversioned guest-agent API at exactly what v0.5.11 served, and the promise that comes with the freeze is that a released 0.5.x SDK keeps working against a 0.6 agent unchanged. Nothing tested that. The descriptor-digest test pins the proto's shape, but a shape can hold while behaviour moves underneath it, and every suite in this repo is edited in the same commit as the code it covers -- so a break is invisible exactly when it matters. `sdk/compat/run-compat-tests.sh <tag>` builds the agent-backed simulator from the current checkout, then checks the SDKs out at `<tag>` with `git worktree` and runs their own suites against it. Old client, new agent, and the only thing crossing between the two trees is the wire protocol. A released client cannot be edited to accommodate a change, which is the property a pinned-in-repo test cannot have. Both tags pass in full today, with an empty skip list in all four languages: the freeze currently holds with no exceptions. Two 0.6.0 changes were expected to need entries and did not -- `EmitEvent` fails with the HTTP 400 the released Python suite already asserted, and `GetQuote`'s TDX-only restriction does not bite a simulator that serves a TDX quote. The skip-list policy is written down where the list lives: an entry must name a sanctioned change and where it is recorded, and a growing list is the failure signal rather than the fix. The simulator lifecycle moves to `sdk/simulator/lifecycle.sh`, shared with `sdk/run-tests.sh`, because the compat runner needs to hold one simulator across several SDK checkouts. Extracting it surfaced a bug in the original: the subshell that starts the simulator is not elided by bash when traps are installed, so `$!` was the subshell rather than the simulator. Cleanup killed the wrapper and left the simulator orphaned, holding its binary open until the next run's build failed with "Text file busy" -- which is what the intermittent stale-socket failures were. `exec` in the subshell fixes it.
1 parent 95f1268 commit 1594816

5 files changed

Lines changed: 502 additions & 66 deletions

File tree

‎.github/workflows/sdk-compat.yaml‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
# SPDX-FileCopyrightText: © 2025 Phala Network <dstack@phala.network>
2+
#
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
name: SDK compatibility
6+
7+
permissions:
8+
contents: read
9+
10+
on:
11+
push:
12+
branches: [next, 'release/**']
13+
pull_request:
14+
branches: [next, 'release/**']
15+
16+
env:
17+
CARGO_TERM_COLOR: always
18+
# Both the current tree and the released tags pin channel "1.92" in
19+
# rust-toolchain.toml. Without this, rustup would treat that as a toolchain
20+
# distinct from the "1.92.0" installed below and re-download it -- along with
21+
# the three cross-compilation targets the pin asks for, none of which the SDK
22+
# suites need. The compat job is about the wire surface, not about
23+
# reproducing each tag's toolchain provisioning.
24+
RUSTUP_TOOLCHAIN: 1.92.0
25+
26+
jobs:
27+
sdk-compat:
28+
name: ${{ matrix.tag }} SDKs vs current agent
29+
runs-on: ubuntu-latest
30+
strategy:
31+
# Each tag is an independent claim; one failing should not hide the other.
32+
fail-fast: false
33+
matrix:
34+
tag: [v0.5.10, v0.5.11]
35+
steps:
36+
- uses: actions/checkout@v5
37+
with:
38+
# The job checks the released SDKs out with `git worktree add <tag>`,
39+
# which needs that tag's commit and its full tree locally. A shallow
40+
# clone plus `fetch-tags` gives the refs but not a guarantee about the
41+
# objects behind them, so take the whole history: the clone is a
42+
# rounding error next to the Rust build in this job.
43+
fetch-depth: 0
44+
45+
- name: Install Rust
46+
uses: dtolnay/rust-toolchain@1.92.0
47+
48+
- name: Install Go
49+
uses: actions/setup-go@v5
50+
with:
51+
# The `go` directive in the released sdk/go/go.mod.
52+
go-version: '1.24'
53+
54+
- name: Install Node
55+
uses: actions/setup-node@v5
56+
with:
57+
# The released sdk/js/package.json asks for node >=18; 20 is what the
58+
# JS SDK release workflow publishes from.
59+
node-version: '20'
60+
61+
- name: Install Python
62+
uses: actions/setup-python@v5
63+
with:
64+
# The released sdk/python/pyproject.toml asks for >=3.10; 3.11 is what
65+
# the Python SDK release workflow builds with. The runner script
66+
# installs PDM if it is missing, the same way sdk/run-tests.sh does.
67+
python-version: '3.11'
68+
69+
- name: Released SDKs against the current agent
70+
run: ./sdk/compat/run-compat-tests.sh ${{ matrix.tag }}

‎sdk/compat/README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# SDK compatibility regression
2+
3+
dstack 0.6.0 froze the unversioned guest-agent API at exactly the surface
4+
v0.5.11 served. `DstackGuest` and `Worker` in
5+
`dstack/guest-agent/rpc/proto/agent_rpc.proto` take no new methods, no new
6+
fields and no renumbering, and every new capability goes to `dstack.guest.v1`
7+
instead. The promise that comes with the freeze is that a released 0.5.x SDK
8+
keeps working, unchanged, against a 0.6 agent.
9+
10+
This directory is what turns that promise into a test.
11+
12+
## What it does
13+
14+
`run-compat-tests.sh <tag>` builds the agent-backed simulator from the **current
15+
checkout**, then checks out the SDKs **as they shipped** at `<tag>` and runs
16+
their own test suites against that simulator.
17+
18+
```
19+
sdk/compat/run-compat-tests.sh v0.5.11
20+
sdk/compat/run-compat-tests.sh v0.5.10 v0.5.11 # one simulator, both tags
21+
```
22+
23+
The released SDKs come from `git worktree add <tag>`, so they are the published
24+
code down to the byte, not a reconstruction. Nothing from the tag's tree is
25+
built into the agent and nothing from the current tree is copied into the SDKs.
26+
The only thing crossing between the two is the wire protocol, which is the
27+
entire subject of the test.
28+
29+
That asymmetry is the point. A test suite pinned inside this repo drifts with
30+
the repo: the assertion gets updated in the same commit that changes the
31+
behaviour, and the break is invisible. A released client cannot be edited to
32+
accommodate a change, so it fails when the surface moves — which is what a real
33+
deployed 0.5.x application would do.
34+
35+
The suites run the same four languages `sdk/run-tests.sh` runs — Rust, Go,
36+
Python, JS — including their purely local tests (compose hashing, env
37+
encryption, signature verification vectors). Those need no agent and should pass
38+
unchanged; they are not skipped just because they are not client calls.
39+
40+
CI runs one tag per matrix job (`.github/workflows/sdk-compat.yaml`). Passing
41+
several tags in one local invocation builds and starts the simulator once and
42+
shares a Cargo target directory across them.
43+
44+
Two things `sdk/run-tests.sh` does are deliberately left out: `pdm run check`
45+
(it lints the released SDK's source with today's ruff and mypy, which says
46+
nothing about the agent's wire surface and fails on tool version drift alone)
47+
and the `no_std` build check (a compile-time property of the old types crate,
48+
with no agent involved).
49+
50+
## The skip list
51+
52+
The script carries a per-language skip list. Every entry names a behaviour
53+
0.6.0 deliberately changed on the frozen surface, with a pointer to where that
54+
decision is recorded — a `CHANGELOG.md` entry, or `docs/guest-api-v1.md`.
55+
56+
**A growing skip list is the failure signal, not the fix.** The list existing at
57+
all is a small admission that the freeze has exceptions; every addition to it
58+
enlarges that admission. When an old suite fails, there are exactly two
59+
outcomes:
60+
61+
1. The failure matches a sanctioned change. Add it, with a comment naming the
62+
change and the record it lives in.
63+
2. It does not. Then the frozen surface has drifted, and the agent is what needs
64+
fixing.
65+
66+
There is no third case where a test is skipped because it is inconvenient. If
67+
you cannot write the justification comment, you are looking at outcome 2.

‎sdk/compat/run-compat-tests.sh‎

Lines changed: 258 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,258 @@
1+
#!/bin/bash
2+
3+
# SPDX-FileCopyrightText: © 2025 Phala Network <dstack@phala.network>
4+
#
5+
# SPDX-License-Identifier: Apache-2.0
6+
7+
# Compatibility regression: run released SDK test suites against a current agent.
8+
#
9+
# dstack 0.6.0 froze the unversioned guest-agent API at exactly what v0.5.11
10+
# served (`DstackGuest` and `Worker` in agent_rpc.proto), so a released 0.5.x
11+
# SDK keeps working against a 0.6 agent unchanged. This script is what makes
12+
# that a testable claim rather than a promise: for each released tag it checks
13+
# out the SDKs *as they shipped* and runs their own test suites against a
14+
# simulator built from the CURRENT tree. Old client, new agent -- any drift in
15+
# the frozen surface fails here.
16+
#
17+
# The simulator is always the current one. The SDKs are always the old ones.
18+
# Nothing from the tag's tree is built into the agent, and nothing from the
19+
# current tree is copied into the SDKs; the only thing crossing between them is
20+
# the wire protocol, which is the whole subject of the test.
21+
#
22+
# Usage: sdk/compat/run-compat-tests.sh <tag> [<tag>...]
23+
# e.g. sdk/compat/run-compat-tests.sh v0.5.11
24+
# sdk/compat/run-compat-tests.sh v0.5.10 v0.5.11 # one simulator, both tags
25+
#
26+
# CI runs one tag per matrix job. Passing several tags locally builds and starts
27+
# the simulator once and shares one Cargo target directory across them.
28+
#
29+
# ---------------------------------------------------------------------------
30+
# Skip-list policy
31+
# ---------------------------------------------------------------------------
32+
#
33+
# Each skip entry below names a behaviour 0.6.0 deliberately changed, with a
34+
# pointer to where that decision is written down. Nothing else belongs there.
35+
#
36+
# A growing skip list is not maintenance: it is the signal that the frozen
37+
# v0.5.11 surface has drifted, which is the one thing this job exists to catch.
38+
# If an old test fails and you cannot point at a CHANGELOG entry or a spec that
39+
# sanctions the change, it is a regression -- fix the agent, not this list.
40+
41+
set -Eeuo pipefail
42+
43+
COMPAT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
44+
SDK_DIR="$(cd "$COMPAT_DIR/.." && pwd -P)"
45+
REPO_ROOT="$(cd "$SDK_DIR/.." && pwd -P)"
46+
47+
# shellcheck source=../simulator/lifecycle.sh
48+
source "$SDK_DIR/simulator/lifecycle.sh"
49+
50+
# Old SDK builds go to a target directory of their own: shared across tags so a
51+
# second tag reuses the first one's dependency build, and kept out of the
52+
# simulator's so the two workspaces neither thrash each other's artifacts nor
53+
# leave the tag's build where `sdk/simulator/build.sh` looks for the binary.
54+
# Applied only to the old Rust suite, never to the simulator build.
55+
COMPAT_CARGO_TARGET_DIR="${COMPAT_CARGO_TARGET_DIR:-$REPO_ROOT/dstack/target/sdk-compat}"
56+
57+
WORKTREES=()
58+
59+
cleanup() {
60+
local worktree
61+
for worktree in "${WORKTREES[@]+"${WORKTREES[@]}"}"; do
62+
git -C "$REPO_ROOT" worktree remove --force "$worktree" 2>/dev/null || true
63+
rm -rf "$worktree"
64+
done
65+
WORKTREES=()
66+
simulator_stop
67+
}
68+
69+
trap 'simulator_print_logs' ERR
70+
trap cleanup EXIT INT TERM
71+
72+
usage() {
73+
echo "usage: ${BASH_SOURCE[0]} <tag> [<tag>...]" >&2
74+
echo " e.g. ${BASH_SOURCE[0]} v0.5.10 v0.5.11" >&2
75+
}
76+
77+
# ---------------------------------------------------------------------------
78+
# Skip lists -- see the policy at the top of this file.
79+
# ---------------------------------------------------------------------------
80+
81+
# All four lists are empty, and that is the result, not an oversight: the
82+
# v0.5.10 and v0.5.11 suites pass in full against the 0.6.0 agent, so the freeze
83+
# currently holds with no exceptions. Two 0.6.0 changes were expected to land
84+
# here and did not:
85+
#
86+
# - `EmitEvent` always fails now (CHANGELOG 0.6.0, Removed: "runtime RTMR3
87+
# events are system-owned"). Rust, Go and JS never tested it. Python's
88+
# `test_emit_event` asserts HTTP 400 whenever DSTACK_SIMULATOR_ENDPOINT is
89+
# set -- the simulator had no RTMR to extend at v0.5.11 either -- and the
90+
# 0.6.0 stub fails with exactly HTTP 400, so the released assertion still
91+
# holds.
92+
# - `GetQuote` is Intel TDX only now (CHANGELOG 0.6.0, Changed). The simulator
93+
# serves a TDX quote, so it answers, which is what the released suites
94+
# assert.
95+
#
96+
# Entries with spaces must be quoted; a bare word list splits on them.
97+
98+
# Rust: `cargo test -- --skip <substring>`, matched against the full test path.
99+
RUST_SKIP=(
100+
)
101+
102+
# Go: `go test -skip <regexp>`, matched against the test name.
103+
GO_SKIP=(
104+
)
105+
106+
# Python: `pytest --deselect <file>::<test>`. Matched as a nodeid prefix, so
107+
# `::test_foo` also deselects `::test_foo_bar` -- name the test exactly.
108+
PYTHON_SKIP=(
109+
)
110+
111+
# JS: vitest has no negative name filter, so the entries are woven into one
112+
# negative-lookahead `--testNamePattern`. They are matched as substrings of the
113+
# full test name, `describe` prefixes included.
114+
JS_SKIP=(
115+
)
116+
117+
run_rust_suite() {
118+
local sdk_root="$1"
119+
local skip_args=()
120+
local pattern
121+
122+
for pattern in "${RUST_SKIP[@]+"${RUST_SKIP[@]}"}"; do
123+
skip_args+=(--skip "$pattern")
124+
done
125+
126+
echo "=== rust ==="
127+
(
128+
cd "$sdk_root/rust"
129+
export CARGO_TARGET_DIR="$COMPAT_CARGO_TARGET_DIR"
130+
cargo test -- --show-output "${skip_args[@]+"${skip_args[@]}"}"
131+
# The examples are client exercises too: they drive the agent end to end
132+
# the way a README reader would.
133+
cargo run --example tappd_client_usage
134+
cargo run --example dstack_client_usage
135+
)
136+
}
137+
138+
run_go_suite() {
139+
local sdk_root="$1"
140+
local skip_args=()
141+
local joined=""
142+
local pattern
143+
144+
for pattern in "${GO_SKIP[@]+"${GO_SKIP[@]}"}"; do
145+
joined+="${joined:+|}$pattern"
146+
done
147+
if [[ -n "$joined" ]]; then
148+
skip_args+=(-skip "$joined")
149+
fi
150+
151+
echo "=== go ==="
152+
(
153+
cd "$sdk_root/go"
154+
go clean -testcache
155+
go test -v "${skip_args[@]+"${skip_args[@]}"}" ./dstack
156+
DSTACK_SIMULATOR_ENDPOINT="$TAPPD_SIMULATOR_ENDPOINT" \
157+
go test -v "${skip_args[@]+"${skip_args[@]}"}" ./tappd
158+
)
159+
}
160+
161+
run_python_suite() {
162+
local sdk_root="$1"
163+
local skip_args=()
164+
local pattern
165+
166+
for pattern in "${PYTHON_SKIP[@]+"${PYTHON_SKIP[@]}"}"; do
167+
skip_args+=(--deselect "$pattern")
168+
done
169+
170+
echo "=== python ==="
171+
(
172+
cd "$sdk_root/python"
173+
if ! command -v pdm >/dev/null 2>&1; then
174+
echo "Installing PDM..."
175+
pip install pdm
176+
fi
177+
pdm install --dev
178+
# `pdm run check` is deliberately not run: it lints the released SDK's
179+
# source with today's ruff and mypy, which says nothing about the agent's
180+
# wire surface and would fail on tool version drift alone.
181+
pdm run pytest "${skip_args[@]+"${skip_args[@]}"}"
182+
)
183+
}
184+
185+
run_js_suite() {
186+
local sdk_root="$1"
187+
local name_args=()
188+
local joined=""
189+
local pattern
190+
191+
for pattern in "${JS_SKIP[@]+"${JS_SKIP[@]}"}"; do
192+
joined+="${joined:+|}$pattern"
193+
done
194+
if [[ -n "$joined" ]]; then
195+
name_args+=(--testNamePattern "^(?!.*(?:$joined))")
196+
fi
197+
198+
echo "=== js ==="
199+
(
200+
cd "$sdk_root/js"
201+
npm install
202+
npx vitest --run "${name_args[@]+"${name_args[@]}"}"
203+
)
204+
}
205+
206+
run_tag() {
207+
local tag="$1"
208+
local worktree
209+
210+
worktree="$(mktemp -d -t "dstack-sdk-compat-${tag}-XXXXXX")"
211+
WORKTREES+=("$worktree")
212+
git -C "$REPO_ROOT" worktree add --detach --quiet "$worktree" "refs/tags/$tag"
213+
214+
echo
215+
echo "############################################################"
216+
echo "# $tag SDKs against the current agent"
217+
echo "# sdks: $worktree/sdk"
218+
echo "# simulator: $DSTACK_SIMULATOR_ENDPOINT"
219+
echo "############################################################"
220+
221+
run_rust_suite "$worktree/sdk"
222+
run_go_suite "$worktree/sdk"
223+
run_python_suite "$worktree/sdk"
224+
run_js_suite "$worktree/sdk"
225+
226+
# Drop it now rather than at exit, so running several tags does not keep a
227+
# full checkout per tag on disk. `cleanup` retries harmlessly at exit.
228+
git -C "$REPO_ROOT" worktree remove --force "$worktree"
229+
rm -rf "$worktree"
230+
231+
echo "--- $tag: all suites passed against the current agent"
232+
}
233+
234+
main() {
235+
if [[ $# -lt 1 ]]; then
236+
usage
237+
exit 2
238+
fi
239+
240+
local tag
241+
for tag in "$@"; do
242+
if ! git -C "$REPO_ROOT" rev-parse --verify --quiet "refs/tags/$tag^{commit}" >/dev/null; then
243+
echo "unknown tag: $tag -- fetch tags first (git fetch --tags)" >&2
244+
exit 1
245+
fi
246+
done
247+
248+
simulator_start
249+
250+
for tag in "$@"; do
251+
run_tag "$tag"
252+
done
253+
254+
echo
255+
echo "compat: $* passed against the agent in $REPO_ROOT"
256+
}
257+
258+
main "$@"

0 commit comments

Comments
 (0)