Skip to content

Commit cea88f3

Browse files
scotwellsclaude
scotwells
and
claude
committed
ci(e2e): run in-cluster federation e2e on GitHub Actions
Replaces the placeholder E2E workflow (a single throwaway Kind cluster running a commented-out make target) with one that stands up the full harness the Taskfile builds and runs the Chainsaw suites against it: three Kind clusters (management/control-plane hosting Karmada, plus the dfw and ord POP cells), the real production kustomize overlays, and the real hub RBAC. This validates the operators as deployed pods authenticating to Karmada as a non-admin identity rather than as an in-process test binary. The job splits provisioning, deploy, and test into separate steps so a failure lands on the phase that broke, and always collects per-cluster operator logs, Karmada component logs, events, and a full kind log export as an artifact when any step fails. Runs on the free ubuntu-latest runner: the harness is engineered for a constrained host (single-replica Karmada, leader election disabled, generous component waits), so a larger paid runner is not needed unless real runs show resource pressure. Triggers only on pull_request and pushes to main rather than every branch push, given the job's cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent fe906df commit cea88f3

1 file changed

Lines changed: 127 additions & 9 deletions

File tree

‎.github/workflows/test-e2e.yml‎

Lines changed: 127 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,44 @@
11
name: E2E Tests
22

3+
# In-cluster federation e2e (issue #149).
4+
#
5+
# Stands up the full local topology the Taskfile harness builds — three Kind
6+
# clusters (one management/control-plane hosting Karmada, two POP cells) with the
7+
# real production kustomize overlays and hub RBAC — then runs the Chainsaw
8+
# suites against it. This exercises the operators as deployed pods authenticating
9+
# to Karmada as a non-admin identity, not as an in-process test binary.
10+
311
on:
412
push:
13+
branches: [main]
514
pull_request:
615

16+
# Cancel a superseded run on the same ref. The e2e job is expensive (three Kind
17+
# clusters + a Karmada control plane), so we don't want stale pushes burning a
18+
# runner. Unlike the cheaper test/lint workflows this only triggers on PRs and
19+
# main pushes rather than every branch push, for the same cost reason.
20+
concurrency:
21+
group: e2e-${{ github.workflow }}-${{ github.ref }}
22+
cancel-in-progress: true
23+
24+
permissions:
25+
contents: read
26+
727
jobs:
828
test-e2e:
929
name: Run on Ubuntu
30+
# ubuntu-latest is 4 vCPU / 16 GB. The harness is deliberately engineered for
31+
# a constrained/busy host — single-replica Karmada, leader election disabled,
32+
# generous 10m component waits, join retries — so this fits without a larger
33+
# (paid) runner. If real runs show OOM or repeated timeouts, switch to a
34+
# larger hosted runner label (e.g. ubuntu-latest-8-cores); that carries a
35+
# billing implication, hence starting on the free tier.
1036
runs-on: ubuntu-latest
37+
# Env standup can spend up to ~20m if both Karmada waits approach their 10m
38+
# ceilings on a slow runner; deploy ~5m; the Chainsaw suites ~15m (the
39+
# referenced-data GC-sweep suite alone floors around 6m). 50m leaves headroom
40+
# over the ~35m expected without letting a genuinely hung run idle too long.
41+
timeout-minutes: 50
1142
steps:
1243
- name: Clone the code
1344
uses: actions/checkout@v4
@@ -17,19 +48,106 @@ jobs:
1748
with:
1849
go-version: '~1.25.0'
1950

20-
- name: Install the latest version of kind
51+
# go-task drives the entire harness (task e2e:env:up / e2e:deploy /
52+
# e2e:test). The action publishes major refs as branches (v1/v2/v3); the
53+
# version input pins the go-task binary itself. repo-token avoids GitHub
54+
# API rate limiting when the action resolves the release.
55+
- name: Install go-task
56+
uses: arduino/setup-task@v3
57+
with:
58+
version: 3.52.0
59+
repo-token: ${{ secrets.GITHUB_TOKEN }}
60+
61+
# Kind is not preinstalled on the runner. Docker, kubectl, and helm already
62+
# are (ubuntu-24.04 image); karmadactl and chainsaw are fetched into ./bin
63+
# by the task tooling. Pinned for reproducibility.
64+
- name: Install kind
2165
run: |
22-
curl -Lo ./kind https://kind.sigs.k8s.io/dl/latest/kind-linux-amd64
66+
curl -sSfLo ./kind https://kind.sigs.k8s.io/dl/v0.32.0/kind-linux-amd64
2367
chmod +x ./kind
2468
sudo mv ./kind /usr/local/bin/kind
69+
kind version
70+
71+
# The Karmada kubeconfig rewrite step (_e2e:karmada:build-kubeconfig) shells
72+
# out to python3 + PyYAML. It is normally present on the runner image; guard
73+
# the case where it is not without tripping over PEP 668.
74+
- name: Ensure PyYAML
75+
run: |
76+
if ! python3 -c "import yaml" 2>/dev/null; then
77+
sudo apt-get update
78+
sudo apt-get install -y python3-yaml
79+
fi
2580
26-
- name: Verify kind installation
27-
run: kind version
81+
# Split into env / deploy / test so a failure lands on the phase that broke
82+
# rather than a single opaque "task e2e:up" step. e2e:env:up == e2e:up minus
83+
# the deploy; the two together are exactly what e2e:up runs.
84+
- name: Provision Kind + Karmada environment
85+
run: task e2e:env:up
2886

29-
- name: Create kind cluster
30-
run: kind create cluster
87+
- name: Build image and deploy operators
88+
run: task e2e:deploy
3189

32-
- name: Running Test e2e
90+
- name: Run Chainsaw e2e suites
91+
run: task e2e:test
92+
93+
# Always capture what the clusters looked like when a step failed. The
94+
# runner is ephemeral so teardown is unnecessary; diagnostics are the only
95+
# thing worth keeping.
96+
- name: Collect diagnostics
97+
if: failure()
3398
run: |
34-
go mod tidy
35-
make test-e2e
99+
set +e
100+
DIAG=tmp/e2e/diagnostics
101+
KDIR=tmp/e2e/kubeconfigs
102+
mkdir -p "$DIAG"
103+
104+
# Host / kind level: container state plus a full per-cluster export
105+
# (kubelet, containerd, and every pod log).
106+
kind get clusters > "$DIAG/kind-clusters.txt" 2>&1
107+
docker ps -a > "$DIAG/docker-ps.txt" 2>&1
108+
for c in compute-control-plane compute-pop-dfw compute-pop-ord; do
109+
kind export logs "$DIAG/kind-$c" --name "$c" 2>&1 | tail -n 2
110+
done
111+
112+
# Per-cluster Kubernetes state + the compute-manager operator logs
113+
# (current and previous, all containers) from every plane.
114+
for kc in control-plane pop-dfw pop-ord karmada; do
115+
cfg="$KDIR/$kc.yaml"
116+
[ -f "$cfg" ] || continue
117+
out="$DIAG/$kc"; mkdir -p "$out"
118+
kubectl --kubeconfig="$cfg" get pods -A -o wide > "$out/pods.txt" 2>&1
119+
kubectl --kubeconfig="$cfg" get events -A --sort-by=.lastTimestamp > "$out/events.txt" 2>&1
120+
kubectl --kubeconfig="$cfg" -n compute-system describe deploy compute-manager \
121+
> "$out/compute-manager-describe.txt" 2>&1
122+
kubectl --kubeconfig="$cfg" -n compute-system logs deploy/compute-manager \
123+
--all-containers --tail=-1 > "$out/compute-manager.log" 2>&1
124+
kubectl --kubeconfig="$cfg" -n compute-system logs deploy/compute-manager \
125+
--all-containers --previous --tail=-1 > "$out/compute-manager-previous.log" 2>&1
126+
done
127+
128+
# Karmada control-plane pods + component logs (they live in the
129+
# management cluster) and the federation view from the Karmada API.
130+
if [ -f "$KDIR/control-plane.yaml" ]; then
131+
kubectl --kubeconfig="$KDIR/control-plane.yaml" -n karmada-system get pods -o wide \
132+
> "$DIAG/karmada-pods.txt" 2>&1
133+
for d in karmada-apiserver karmada-controller-manager karmada-scheduler; do
134+
kubectl --kubeconfig="$KDIR/control-plane.yaml" -n karmada-system logs deploy/$d \
135+
--tail=-1 > "$DIAG/karmada-$d.log" 2>&1
136+
done
137+
fi
138+
if [ -f "$KDIR/karmada.yaml" ]; then
139+
kubectl --kubeconfig="$KDIR/karmada.yaml" get clusters -o wide \
140+
> "$DIAG/karmada-clusters.txt" 2>&1
141+
kubectl --kubeconfig="$KDIR/karmada.yaml" get workloaddeployments -A -o wide \
142+
> "$DIAG/karmada-workloaddeployments.txt" 2>&1
143+
fi
144+
echo "Diagnostics collected under $DIAG"
145+
146+
- name: Upload diagnostics
147+
if: failure()
148+
uses: actions/upload-artifact@v4
149+
with:
150+
name: e2e-diagnostics
151+
path: tmp/e2e/diagnostics
152+
retention-days: 7
153+
if-no-files-found: warn

0 commit comments

Comments
 (0)