Skip to content

Commit 6c9ccad

Browse files
committed
gha: capture per-pod diagnostics for sig-network conntrack failures
The sig-network Conntrack "should be able to preserve UDP traffic ... NodePort" spec has been failing at conntrack.go:214 "Failed to connect to backend 2" while the backend-1 check passed, which means Cilium already forwarded and replied to the UDP NodePort flow correctly. The real trigger is the client probe pod (pod-client) sitting in CrashLoopBackOff during the one-minute backend-2 poll window, so its logs freeze and the substring poll deterministically times out. We could not prove that from CI: the run uses --disable-log-dump=true and the test namespace is torn down in AfterEach, so pod-client's lastState.terminated exit code and reason, and the previous container stdout that would show whether the nc loop ran to completion, were gone by the time we looked. This adds instrumentation instead of a behavioural change, because a genuine backend-2 datapath regression would surface at the same line and masking it is unacceptable. I pass --delete-namespace-on-failure=false to the ginkgo run so only failed specs' namespaces survive past AfterEach, and in the post-test gathering step, before the kind export, I capture pods -o wide and -o yaml, sorted events, and for every pod left in a conntrack/services/nettest/network namespace a describe plus current and --previous container logs. The next occurrence will show pod-client's exit code and whether the 3000-iteration nc loop completed, which is what tells a client-pod crash apart from a datapath fault. AIL:3 Signed-off-by: André Martins <andre@cilium.io>
1 parent 2deb5d2 commit 6c9ccad

1 file changed

Lines changed: 34 additions & 1 deletion

File tree

.github/workflows/k8s-kind-network-e2e.yaml

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -296,7 +296,8 @@ jobs:
296296
--provider=local \
297297
--dump-logs-on-failure=true \
298298
--report-dir=${E2E_REPORT_DIR} \
299-
--disable-log-dump=true
299+
--disable-log-dump=true \
300+
--delete-namespace-on-failure=false
300301
301302
# Sequentially kill kube-apiserver pods to verify cilium agent pods fail over to an active instance.
302303
- name: Test kube-apiserver high availability
@@ -349,6 +350,38 @@ jobs:
349350
if: ${{ !success() && steps.install-cilium.outcome != 'skipped' }}
350351
run: |
351352
mkdir -p ./_artifacts/logs
353+
354+
# The sig-network run keeps failed specs' namespaces alive
355+
# (--delete-namespace-on-failure=false), so capture per-pod state
356+
# before the kind export tears the cluster down. This is where the
357+
# sig-network Conntrack UDP NodePort failure lives: the client probe
358+
# pod (pod-client) has been observed crash-looping during the
359+
# backend poll window, and namespace teardown plus --disable-log-dump
360+
# previously destroyed the evidence (lastState.terminated exit code
361+
# /reason and the --previous container stdout that shows whether the
362+
# nc loop ran to completion) needed to tell a client-pod crash apart
363+
# from a real datapath regression.
364+
kubectl get pods -A -o wide > ./_artifacts/logs/pods-wide.txt 2>&1
365+
kubectl get pods -A -o yaml > ./_artifacts/logs/pods.yaml 2>&1
366+
kubectl get events -A --sort-by=.lastTimestamp > ./_artifacts/logs/events.txt 2>&1
367+
368+
# For every pod left behind in a sig-network test namespace, dump the
369+
# describe output, current logs and previous-container logs so a
370+
# future occurrence is fully diagnosable.
371+
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
372+
case "$ns" in
373+
conntrack-*|services-*|nettest-*|network-*)
374+
nsdir="./_artifacts/logs/${ns}"
375+
mkdir -p "$nsdir"
376+
for pod in $(kubectl -n "$ns" get pods -o jsonpath='{.items[*].metadata.name}'); do
377+
kubectl -n "$ns" describe pod "$pod" > "${nsdir}/${pod}.describe.txt" 2>&1
378+
kubectl -n "$ns" logs "$pod" --all-containers=true > "${nsdir}/${pod}.log" 2>&1
379+
kubectl -n "$ns" logs "$pod" --all-containers=true --previous > "${nsdir}/${pod}.previous.log" 2>&1
380+
done
381+
;;
382+
esac
383+
done
384+
352385
/usr/local/bin/kind export logs --name ${{ env.cluster_name }} --verbosity=3 ./_artifacts/logs
353386
shell: bash {0} # Disable default fail-fast behaviour so that all commands run independently
354387

0 commit comments

Comments
 (0)