Skip to content

Commit 0fc0839

Browse files
committed
Fix: EXIT trap exit code propagation on bash 5.1
The openstack cron job (builder-openstack-cron) has been failing consistently since the build-node migration from CentOS 8 (bash 4.4) to Ubuntu 22.04 (bash 5.1) in releng/builder commit f601068e8 (Chore: Replace CentOS build-node refs with Ubuntu). Problem: The cron job runs 7 cleanup scripts sequentially via conditional build steps: k8s clusters, stacks, servers, ports, volumes, image-protect, and image-cleanup. After the migration, the build consistently failed at the ports cleanup step with: No orphaned ports to process. Build step 'Conditional step (single)' marked build as failure All subsequent steps (volumes, image-protect, image-cleanup) were skipped due to the BuildStepRunner$Fail runner aborting the build. Root Cause: Bash 5.1 (shipped with Ubuntu 22.04) has a known behavior where when 'set -e' (errexit) is active, a failing command inside an EXIT trap causes the trap's non-zero exit code to propagate as the script's final exit code, overriding an explicit 'exit 0'. The _rmtemp() EXIT trap function used this pattern: _rmtemp() { if [ -f "$tmpfile" ]; then rm -f "$tmpfile" fi } When the temp file does not exist (normal case after cleanup), the 'if [ -f ]' test returns false, making the if-statement's exit code 1. Bash 5.1 with 'set -e' propagates this as the script's final exit code, causing Jenkins to see a non-zero return and marking the build step as FAILURE. Bash 4.4 (CentOS 8) does not exhibit this behavior — it preserves the original 'exit 0' regardless of trap exit codes, which is why the job worked before the migration. This only affected scripts with 'set -eu -o pipefail': - openstack-cleanup-orphaned-ports.sh (set -eu -o pipefail) - openstack-cleanup-orphaned-objects.sh (set -eu -o pipefail) Scripts without 'set -e' were unaffected: - openstack-cleanup-orphaned-k8s-clusters.sh (no set -e) - openstack-cleanup-orphaned-stacks.sh (set -x only) - openstack-cleanup-orphaned-servers.sh (no set -e) This explains why the cron job always failed at the ports step (the first script in the sequence with 'set -eu'). Fix: Replace the 'if [ -f ] / rm' pattern with 'rm -f || true' to ensure the EXIT trap always returns exit code 0 regardless of whether the temp file exists. The 'rm -f' flag already suppresses errors for non-existent files, and '|| true' provides a safety net for any unexpected failures. Stderr is redirected to /dev/null to suppress any spurious error messages. Investigation: - Build 77811 (PASS): ran on prd-centos8-builder-2c-2g (bash 4.4) - Build 77812 (FAIL): first build on prd-ubuntu2204-builder-2c-2g (bash 5.1) after the CentOS-to-Ubuntu migration - All builds from 77812 to 77971+ consistently fail at ports step - Confirmed via S3 console logs comparison and binary search References: - https://stackoverflow.com/questions/41619629 - https://unix.stackexchange.com/questions/667368 Issue: RELENG-5951 Change-Id: Id09def17f0dbdbabebea0f96b8ef9ebc43778f93 Signed-off-by: Anil Belur <abelur@linuxfoundation.org>
1 parent 7f53049 commit 0fc0839

3 files changed

Lines changed: 15 additions & 8 deletions

File tree

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
fixes:
3+
- |
4+
Fix openstack cron job EXIT trap causing false build failures on
5+
bash 5.1 (Ubuntu 22.04). When ``set -e`` is active and a command
6+
in the EXIT trap returns non-zero, bash 5.1 propagates the trap's
7+
failure exit code instead of preserving the original ``exit 0``.
8+
The ``_rmtemp`` trap used an ``if [ -f ]`` pattern that returned
9+
exit code 1 when the temp file didn't exist, causing the cron job
10+
to fail despite all cleanup scripts completing successfully.
11+
Changed to ``rm -f || true`` to ensure the trap always returns 0.

shell/openstack-cleanup-orphaned-objects.sh

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,10 +72,8 @@ _cleanup()
7272

7373
_rmtemp()
7474
{
75-
if [ -f "$tmpfile" ]; then
76-
# Removes temporary file on script exit
77-
rm -f "$tmpfile"
78-
fi
75+
# Removes temporary file on script exit
76+
rm -f "$tmpfile" 2>/dev/null || true
7977
}
8078

8179
trap _rmtemp EXIT

shell/openstack-cleanup-orphaned-ports.sh

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -57,10 +57,8 @@ _cleanup()
5757

5858
_rmtemp()
5959
{
60-
if [ -f "$tmpfile" ]; then
61-
# Removes temporary file on script exit
62-
rm -f "$tmpfile"
63-
fi
60+
# Removes temporary file on script exit
61+
rm -f "$tmpfile" 2>/dev/null || true
6462
}
6563

6664
trap _rmtemp EXIT

0 commit comments

Comments
 (0)