Commit 0fc0839
committed
Fix: EXIT trap exit code propagation on bash 5.1
The openstack cron job (builder-openstack-cron) has been failing
consistently since the build-node migration from CentOS 8 (bash 4.4)
to Ubuntu 22.04 (bash 5.1) in releng/builder commit f601068e8
(Chore: Replace CentOS build-node refs with Ubuntu).
Problem:
The cron job runs 7 cleanup scripts sequentially via conditional
build steps: k8s clusters, stacks, servers, ports, volumes,
image-protect, and image-cleanup. After the migration, the build
consistently failed at the ports cleanup step with:
No orphaned ports to process.
Build step 'Conditional step (single)' marked build as failure
All subsequent steps (volumes, image-protect, image-cleanup) were
skipped due to the BuildStepRunner$Fail runner aborting the build.
Root Cause:
Bash 5.1 (shipped with Ubuntu 22.04) has a known behavior where
when 'set -e' (errexit) is active, a failing command inside an
EXIT trap causes the trap's non-zero exit code to propagate as
the script's final exit code, overriding an explicit 'exit 0'.
The _rmtemp() EXIT trap function used this pattern:
_rmtemp() {
if [ -f "$tmpfile" ]; then
rm -f "$tmpfile"
fi
}
When the temp file does not exist (normal case after cleanup),
the 'if [ -f ]' test returns false, making the if-statement's
exit code 1. Bash 5.1 with 'set -e' propagates this as the
script's final exit code, causing Jenkins to see a non-zero
return and marking the build step as FAILURE.
Bash 4.4 (CentOS 8) does not exhibit this behavior — it
preserves the original 'exit 0' regardless of trap exit codes,
which is why the job worked before the migration.
This only affected scripts with 'set -eu -o pipefail':
- openstack-cleanup-orphaned-ports.sh (set -eu -o pipefail)
- openstack-cleanup-orphaned-objects.sh (set -eu -o pipefail)
Scripts without 'set -e' were unaffected:
- openstack-cleanup-orphaned-k8s-clusters.sh (no set -e)
- openstack-cleanup-orphaned-stacks.sh (set -x only)
- openstack-cleanup-orphaned-servers.sh (no set -e)
This explains why the cron job always failed at the ports step
(the first script in the sequence with 'set -eu').
Fix:
Replace the 'if [ -f ] / rm' pattern with 'rm -f || true' to
ensure the EXIT trap always returns exit code 0 regardless of
whether the temp file exists. The 'rm -f' flag already suppresses
errors for non-existent files, and '|| true' provides a safety
net for any unexpected failures. Stderr is redirected to /dev/null
to suppress any spurious error messages.
Investigation:
- Build 77811 (PASS): ran on prd-centos8-builder-2c-2g (bash 4.4)
- Build 77812 (FAIL): first build on prd-ubuntu2204-builder-2c-2g
(bash 5.1) after the CentOS-to-Ubuntu migration
- All builds from 77812 to 77971+ consistently fail at ports step
- Confirmed via S3 console logs comparison and binary search
References:
- https://stackoverflow.com/questions/41619629
- https://unix.stackexchange.com/questions/667368
Issue: RELENG-5951
Change-Id: Id09def17f0dbdbabebea0f96b8ef9ebc43778f93
Signed-off-by: Anil Belur <abelur@linuxfoundation.org>1 parent 7f53049 commit 0fc0839
3 files changed
Lines changed: 15 additions & 8 deletions
File tree
- releasenotes/notes
- shell
Lines changed: 11 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
| 75 | + | |
| 76 | + | |
79 | 77 | | |
80 | 78 | | |
81 | 79 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
| 60 | + | |
| 61 | + | |
64 | 62 | | |
65 | 63 | | |
66 | 64 | | |
| |||
0 commit comments