Skip to content

Commit 6ee77b7

Browse files
iplay88keysEItanya
iplay88keys
and
EItanya
authored
Add database upgrade and rollback compatibility tests (#2084)
## Description Closes: #1637 Adds automated coverage that a Helm upgrade from a prior release to the current build is safe: migrations apply cleanly, existing data survives, the resulting schema matches a fresh install, and the migrations reverse cleanly. Also covers the rolling-deploy window where prior-release pods serve against an already-migrated schema, and — at each point in the upgrade/rollback cycle — drives a real agent invocation through the controller so kagent's own query paths are exercised, not just raw SQL. ## Tests - **Upgrade round-trip** (`TestUpgrade`, gated by `RUN_UPGRADE_TESTS`): installs the prior release, seeds representative data, upgrades to the current build, then asserts the controller rolls out without crashing, migrations reach the target version cleanly, the seeded data survives, the upgraded schema is identical to a clean install, and reversing the migrations restores the prior schema with data intact. It then redeploys the previous release over the migrated schema to confirm the old binary boots and serves rather than crash-looping (ahead-schema tolerance). - **Rolling compatibility** (`TestRollingUpgradeCompatibility`, gated by `RUN_ROLLING_UPGRADE_TESTS`): runs the upgrade with two controller replicas and verifies a prior-release pod can still read and write while the new image rolls out. It runs regardless of whether migrations changed — the image swap alone can break a rolling deploy — and additionally exercises the old-code/new-schema window when the build adds migrations. - **Version-matched invocation**: at each state — post-upgrade, old-code-against-the-new-schema, and post-rollback — a representative invoke e2e slice (`TestE2EInvokeInlineAgent`) runs from the tree whose version matches the serving controller (the previous-release slices from a git worktree at that release's tag), so real deploy/invoke/read paths are exercised end to end. Self-skips when the harness (mock-LLM host / previous-release worktree) isn't set up, so a bare `go test` still runs the DB round-trip. ## Targets Upgrade-from targets are chosen per branch, mirroring how releases ship (a CI matrix; a leg is skipped when it doesn't apply): - on `main` (building the next minor): the previous minor line's latest patch. - on a release branch (building a patch): the same line's last patch, and the previous minor's latest. The resolved upgrade-from version is printed before the build. Override locally with `UPGRADE_FROM_VERSION`. ## Running locally Requires a kind cluster, then: ``` make create-kind-cluster make run-upgrade-tests ``` or ``` make create-kind-cluster make run-rolling-upgrade-tests ``` A run leaves the database migrated, so recreate the cluster between runs. ## Notes - The previous install pins the bundled Postgres image to whatever the upgrade-from release's own install target shipped, so the baseline matches how that release actually runs rather than a hardcoded guess. - Query-level backward compatibility (a prior release's queries against the new schema) is covered separately by a static check in #2085. --------- Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io> Co-authored-by: Eitan Yarmush <eitan.yarmush@solo.io>
1 parent 6a241b7 commit 6ee77b7

10 files changed

Lines changed: 1528 additions & 4 deletions

File tree

‎.claude/skills/kagent-dev/references/database-migrations.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ Files must follow `NNNNNN_description.up.sql` / `NNNNNN_description.down.sql` wi
133133

134134
Every `.up.sql` must have a corresponding `.down.sql` that exactly reverses it. Down migrations are used for rollbacks and by automatic rollback on migration failure. They must be **idempotent** — the two-track rollback logic (roll back core if vector fails) may call them more than once in failure scenarios.
135135

136-
A down file that never runs is a down file you cannot trust. There are no up-only migrations — a working down has shipped with every migration since the golang-migrate adoption. Exercising every migration up → down → up against the real migration set, to prove the reversal rather than assume it, is a *Target — not yet enforced* (see [Upgrade and rollback testing](#upgrade-and-rollback-testing)).
136+
A down file that never runs is a down file you cannot trust. There are no up-only migrations — a working down has shipped with every migration since the golang-migrate adoption. The reversal is proven, not assumed: the upgrade round-trip applies `HEAD`'s migrations over a prior release and then reverses them back, asserting the reverted schema matches a clean install of that release and that seeded data survives (see [Upgrade and rollback testing](#upgrade-and-rollback-testing)).
137137

138138
## One Linear History
139139

@@ -205,9 +205,9 @@ These tests catch policy violations at PR time without needing a running databas
205205

206206
## Upgrade and rollback testing
207207

208-
Static analysis covers file *content*; round-trip tests cover *behavior* against a real Postgres. Beyond `runner_test.go` (rollback and concurrency), release-to-release coverage makes the rollback promise real.
208+
Static analysis covers file *content*; round-trip tests cover *behavior* against a real Postgres. Beyond `runner_test.go` (rollback and concurrency), release-to-release tests make the rollback promise real.
209209

210-
**Previous-minor round-trip** (*Target — not yet enforced*). Seed a database at the previous minor's latest release with representative data, apply migrations up to `HEAD`, and assert the schema matches a clean `HEAD` install and the data survives; then reverse to the previous minor and assert the schema matches a clean previous-minor install and the data survives. This exercises every changed down file rather than only reviewing it.
210+
**Previous-release round-trip** (enforced by `TestUpgrade`, run by the `upgrade-tests` CI job). Seed a database at a prior release with representative data, apply migrations up to `HEAD`, and assert the controller rolls out without crashing, the schema matches a clean `HEAD` install, and the data survives; then reverse the migrations back to the prior release and assert the schema matches a clean install of that release and the data survives. It runs against two prior versions — the latest release reachable from `HEAD` and the previous stable line's latest patch (the `release/vX.Y.x` tip) — and `TestRollingUpgradeCompatibility` (the `rolling-upgrade-tests` job) additionally exercises the old-code/new-schema window, with the prior release's controller serving while `HEAD`'s migrations are applied.
211211

212212
**Query-level backward compatibility.** A static check — `scripts/check-query-contraction.sh`, run by the `query-contraction-check` CI job — compiles a previous release's sqlc queries against the `HEAD` schema and fails if a migration dropped, renamed, or retyped a column or table an older query still reads. It catches column/table/type-shape contraction with no database, against two prior versions: the latest release reachable from `HEAD` and the previous stable line's latest patch (the `release/vX.Y.x` tip, via `scripts/prev-stable-version.sh`). The fuller property — running the previous minor's whole database *test suite* against a `HEAD`-migrated schema, which also covers semantic breaks a query still compiles against — remains a *Target — not yet enforced*.
213213

‎.github/actions/upgrade-test-setup/action.yaml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
name: Upgrade Test Setup
2+
3+
description: >-
4+
Shared prelude for the upgrade-tests and rolling-upgrade-tests jobs: resolve
5+
the upgrade-from version (with the prev-stable == adjacent skip) and bring up
6+
the build/cluster toolchain. Exposes the resolved version and skip flag so the
7+
caller can gate its test step. The caller MUST run actions/checkout (with
8+
fetch-depth: 0 + fetch-tags) before this action — a local action is loaded
9+
from the checked-out workspace and the version resolvers need full history.
10+
11+
inputs:
12+
upgrade-from:
13+
description: 'Which release to upgrade from: "adjacent" or "prev-stable".'
14+
required: true
15+
16+
outputs:
17+
skip:
18+
description: '"true" when this leg is redundant (prev-stable == adjacent) and the caller should skip its test step.'
19+
value: ${{ steps.resolve.outputs.skip }}
20+
version:
21+
description: The resolved upgrade-from version (empty when skip is true).
22+
value: ${{ steps.resolve.outputs.version }}
23+
24+
runs:
25+
using: "composite"
26+
steps:
27+
# The caller must run actions/checkout (fetch-depth: 0 + fetch-tags) before
28+
# this action: a local action is loaded from the checked-out workspace, so
29+
# its files don't exist on the runner until checkout has run.
30+
- name: Resolve upgrade-from version
31+
id: resolve
32+
shell: bash
33+
run: |
34+
ADJ="$(./scripts/upgrade-from-version.sh)"
35+
base_ref="${GITHUB_BASE_REF:-${GITHUB_REF_NAME:-}}"
36+
is_release_line=false
37+
if [[ "$base_ref" =~ ^release/v[0-9]+\.[0-9]+\.x$ ]]; then
38+
is_release_line=true
39+
fi
40+
41+
if [ "${{ inputs.upgrade-from }}" = "prev-stable" ]; then
42+
# prev-stable = the previous minor line's latest patch (the rollback
43+
# floor). On main this resolves to the newest release line overall
44+
# (e.g. the 0.9.x tip); on a release branch, to the line below it
45+
# (skipped when none exists).
46+
V="$(./scripts/prev-stable-version.sh)"
47+
if [ -z "$V" ]; then
48+
echo "no stable line below the current line; skipping prev-stable leg."
49+
echo "skip=true" >> "$GITHUB_OUTPUT"
50+
exit 0
51+
fi
52+
if [ "$V" = "$ADJ" ]; then
53+
echo "prev-stable ($V) == adjacent; skipping (covered by the adjacent leg)."
54+
echo "skip=true" >> "$GITHUB_OUTPUT"
55+
exit 0
56+
fi
57+
else
58+
# adjacent = the same-line previous patch, i.e. the release just before
59+
# the one being built on the SAME line. It is only meaningful when
60+
# building a patch on a release branch. On main we are building the next
61+
# MINOR, which has no same-line predecessor (git-describe would report a
62+
# stale patch nobody upgrades from — the newest patch lives on the
63+
# release branch, covered by prev-stable), so skip this leg there.
64+
if [ "$is_release_line" != "true" ]; then
65+
echo "base '$base_ref' is not a release line (building the next minor); skipping the adjacent leg (only prev-stable is meaningful)."
66+
echo "skip=true" >> "$GITHUB_OUTPUT"
67+
exit 0
68+
fi
69+
V="$ADJ"
70+
fi
71+
echo "version=$V" >> "$GITHUB_OUTPUT"
72+
echo "=== Upgrade test (${{ inputs.upgrade-from }} leg): will upgrade FROM v$V TO the current build — building images next ==="
73+
echo "::notice title=Upgrade from::v$V (${{ inputs.upgrade-from }} leg) -> current build"
74+
- name: Initialize Environment
75+
if: steps.resolve.outputs.skip != 'true'
76+
uses: ./.github/actions/initialize-environment
77+
- name: Allow unprivileged user namespaces
78+
if: steps.resolve.outputs.skip != 'true'
79+
shell: bash
80+
run: |
81+
sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 || true
82+
- name: Set up QEMU
83+
if: steps.resolve.outputs.skip != 'true'
84+
uses: docker/setup-qemu-action@v4
85+
with:
86+
platforms: linux/amd64,linux/arm64
87+
- name: Set up Docker Buildx
88+
if: steps.resolve.outputs.skip != 'true'
89+
uses: docker/setup-buildx-action@v4
90+
with:
91+
# Builder name/version match the Makefile (BUILDKIT_VERSION) and the other
92+
# workflows (e.g. tag.yaml); kept as literals for consistency.
93+
name: kagent-builder-v0.23.0
94+
version: v0.23.0
95+
platforms: linux/amd64,linux/arm64
96+
use: "true"
97+
driver-opts: network=host
98+
- name: Set up Helm
99+
if: steps.resolve.outputs.skip != 'true'
100+
uses: azure/setup-helm@v5.0.0
101+
with:
102+
version: v3.18.0
103+
- name: Install Kind
104+
if: steps.resolve.outputs.skip != 'true'
105+
uses: helm/kind-action@ef37e7f390d99f746eb8b610417061a60e82a6cc
106+
with:
107+
install_only: true
108+
- name: Create Kind cluster
109+
if: steps.resolve.outputs.skip != 'true'
110+
shell: bash
111+
run: |
112+
make create-kind-cluster

‎.github/workflows/ci.yaml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,104 @@ jobs:
167167
- name: Previous-release queries vs current schema
168168
run: make -C go check-query-contraction
169169

170+
upgrade-tests:
171+
needs:
172+
- setup
173+
env:
174+
VERSION: v0.0.1-test
175+
runs-on: ubuntu-latest
176+
strategy:
177+
fail-fast: false
178+
matrix:
179+
# prev-stable: the previous minor line's latest patch — runs on every base
180+
# (on main it resolves to the newest release line, e.g. the 0.9.x tip).
181+
# adjacent: the same-line previous patch — only meaningful on a release
182+
# branch (building a patch); skipped on main, which builds a new minor
183+
# with no same-line predecessor. See upgrade-test-setup resolve step.
184+
upgrade-from: [adjacent, prev-stable]
185+
steps:
186+
- name: Checkout repository
187+
uses: actions/checkout@v6
188+
with:
189+
# Full history + tags so the version resolvers can derive the
190+
# upgrade-from release, and so the local action below is on disk.
191+
fetch-depth: 0
192+
fetch-tags: true
193+
- name: Prepare upgrade test environment
194+
id: prep
195+
uses: ./.github/actions/upgrade-test-setup
196+
with:
197+
upgrade-from: ${{ matrix.upgrade-from }}
198+
- name: Run upgrade tests
199+
if: steps.prep.outputs.skip != 'true'
200+
env:
201+
OPENAI_API_KEY: fake
202+
BUILDX_BUILDER_NAME: ${{ env.BUILDX_BUILDER_NAME }}
203+
KAGENT_HELM_EXTRA_ARGS: --cleanup-on-fail=false
204+
DOCKER_BUILD_ARGS: >-
205+
--cache-from=type=gha,scope=${{ needs.setup.outputs.cache-key }}-e2e
206+
--cache-from=type=gha,scope=${{ env.CACHE_KEY_PREFIX }}-main-e2e
207+
--platform=linux/amd64
208+
--push
209+
run: |
210+
make run-upgrade-tests UPGRADE_FROM_VERSION="${{ steps.prep.outputs.version }}"
211+
- name: fail print info
212+
if: failure() && steps.prep.outputs.skip != 'true'
213+
run: |
214+
echo "::error::Failed to run upgrade tests"
215+
kubectl describe pods -n kagent
216+
kubectl get events -n kagent
217+
kubectl logs -n kagent deployment/kagent-controller || true
218+
219+
rolling-upgrade-tests:
220+
needs:
221+
- setup
222+
env:
223+
VERSION: v0.0.1-test
224+
runs-on: ubuntu-latest
225+
strategy:
226+
fail-fast: false
227+
matrix:
228+
# prev-stable: the previous minor line's latest patch — runs on every base
229+
# (on main it resolves to the newest release line, e.g. the 0.9.x tip).
230+
# adjacent: the same-line previous patch — only meaningful on a release
231+
# branch (building a patch); skipped on main, which builds a new minor
232+
# with no same-line predecessor. See upgrade-test-setup resolve step.
233+
upgrade-from: [adjacent, prev-stable]
234+
steps:
235+
- name: Checkout repository
236+
uses: actions/checkout@v6
237+
with:
238+
# Full history + tags so the version resolvers can derive the
239+
# upgrade-from release, and so the local action below is on disk.
240+
fetch-depth: 0
241+
fetch-tags: true
242+
- name: Prepare upgrade test environment
243+
id: prep
244+
uses: ./.github/actions/upgrade-test-setup
245+
with:
246+
upgrade-from: ${{ matrix.upgrade-from }}
247+
- name: Run rolling upgrade tests
248+
if: steps.prep.outputs.skip != 'true'
249+
env:
250+
OPENAI_API_KEY: fake
251+
BUILDX_BUILDER_NAME: ${{ env.BUILDX_BUILDER_NAME }}
252+
KAGENT_HELM_EXTRA_ARGS: --cleanup-on-fail=false
253+
DOCKER_BUILD_ARGS: >-
254+
--cache-from=type=gha,scope=${{ needs.setup.outputs.cache-key }}-e2e
255+
--cache-from=type=gha,scope=${{ env.CACHE_KEY_PREFIX }}-main-e2e
256+
--platform=linux/amd64
257+
--push
258+
run: |
259+
make run-rolling-upgrade-tests UPGRADE_FROM_VERSION="${{ steps.prep.outputs.version }}"
260+
- name: fail print info
261+
if: failure() && steps.prep.outputs.skip != 'true'
262+
run: |
263+
echo "::error::Failed to run rolling upgrade tests"
264+
kubectl describe pods -n kagent
265+
kubectl get events -n kagent
266+
kubectl logs -n kagent deployment/kagent-controller || true
267+
170268
go-unit-tests:
171269
runs-on: ubuntu-latest
172270
steps:

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -213,4 +213,7 @@ file::memory:*
213213

214214
## Test certificates
215215
python/packages/kagent-adk/tests/fixtures/certs/*.pem
216-
python/packages/kagent-adk/tests/fixtures/certs/*.srl
216+
python/packages/kagent-adk/tests/fixtures/certs/*.srl
217+
218+
## Upgrade test: previous-release worktree (created by `make run-upgrade-tests`)
219+
.upgrade-prev/

0 commit comments

Comments
 (0)