sys rootless test failed: podman healthcheck #27906
Description
Issue Description
Hello,
It looks like a test regression I found during working on a PR: sys podman fedora-43 rootless host test failed. I've checked other PRs and found the same error.
I tried reproduce it locally with both binary installed podman and one from my local repo I used to prepare my PR.
Steps to reproduce the issue
Steps to reproduce the issue
- export PODMAN=/usr/bin/podman;
- export QUADLET=/usr/libexec/podman/quadlet
- hack/bats --rootless test/system/220-healthcheck.bats
or it is possible to run locally built podman binaries:
- export PODMAN=/usr/bin/podman;
- export QUADLET=/usr/libexec/podman/quadlet
- hack/bats --rootless test/system/220-healthcheck.bats
Describe the results you received
$ hack/bats --rootless test/system/220-healthcheck.bats
--------------------------------------------------
$ bats test/system/220-healthcheck.bats
✗ podman healthcheck
(in test file test/system/220-healthcheck.bats, line 102)
`cidmatch=$(grep "$cid" <<<"$output")' failed with status 127
[19:32:45.041223951] $ /usr/bin/podman run -d --name c-h- --health-cmd /home/podman/healthcheck --health-interval 1s --health-retries 3 --health-on-failure=kill --health-startup-cmd /home/podman/healthcheck --health-startup-interval 1s quay.io/libpod/testimage:20241011 /home/podman/pause
[19:32:45.389397916] 13d8c3796352e7ef776ffde46fad3c87adca4be371177b11d3d41e794907232e
[19:32:45.399995407] $ /usr/bin/podman inspect c-h- --format {{.Config.HealthcheckOnFailureAction}}
[19:32:45.459968817] kill
[19:32:45.475569608] $ /usr/bin/podman inspect c-h- --format {{.Config.StartupHealthCheck.Test}}
[19:32:45.538683608] [CMD-SHELL /home/podman/healthcheck]
[19:32:45.559405116] $ /usr/bin/podman healthcheck run c-h-
[19:32:45.653386879] $ /usr/bin/podman events --filter container=c-h- --filter event=health_status --since 2026-01-14T19:32:45,549400706+02:00 --stream=false --format {{.HealthStatus}}
[19:32:46.340051211] healthy
healthy
[19:32:46.369313924] $ /usr/bin/podman inspect --format {{json .State.Healthcheck}} c-h-
[19:32:46.440658485] {"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2026-01-14T19:32:45.551225648+02:00","End":"2026-01-14T19:32:45.596771322+02:00","ExitCode":0,"Output":"Life is Good on stdout\nLife is Good on stderr\n"},{"Start":"2026-01-14T19:32:45.615497749+02:00","End":"2026-01-14T19:32:45.630225789+02:00","ExitCode":0,"Output":"Life is Good on stdout\nLife is Good on stderr\n"}]}
[19:32:46.508366560] $ /usr/bin/podman exec c-h- touch /uh-oh
[19:32:46.638356063] $ /usr/bin/podman events --filter container=c-h- --filter event=health_status --since 2026-01-14T19:32:46,500468355+02:00 --stream=false --format {{.HealthStatus}}
[19:32:47.719928692] $ /usr/bin/podman events --filter container=c-h- --filter event=health_status --since 2026-01-14T19:32:46,500468355+02:00 --stream=false --format {{.HealthStatus}}
[19:32:47.793873497] healthy
[19:32:47.813660016] $ /usr/bin/podman inspect --format {{json .State.Healthcheck}} c-h-
[19:32:47.868072765] {"Status":"healthy","FailingStreak":1,"Log":[{"Start":"2026-01-14T19:32:45.551225648+02:00","End":"2026-01-14T19:32:45.596771322+02:00","ExitCode":0,"Output":"Life is Good on stdout\nLife is Good on stderr\n"},{"Start":"2026-01-14T19:32:45.615497749+02:00","End":"2026-01-14T19:32:45.630225789+02:00","ExitCode":0,"Output":"Life is Good on stdout\nLife is Good on stderr\n"},{"Start":"2026-01-14T19:32:46.69213589+02:00","End":"2026-01-14T19:32:46.71283625+02:00","ExitCode":1,"Output":"Uh-oh on stdout!\nUh-oh on stderr!\n"}]}
# [teardown]
...
$
Describe the results you expected
All following tests should have finished successfully:
$ hack/bats --rootless test/system/220-healthcheck.bats
podman info output
$ podman info
host:
arch: amd64
buildahVersion: 1.40.1
cgroupControllers:
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.13-1.fc41.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.13, commit: '
cpuUtilization:
idlePercent: 86.27
systemPercent: 4.63
userPercent: 9.1
cpus: 8
databaseBackend: sqlite
distribution:
distribution: fedora
variant: workstation
version: "41"
eventLogger: journald
freeLocks: 1025
hostname: arsenal-laptop
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 6.14.11-200.fc41.x86_64
linkmode: dynamic
logDriver: journald
memFree: 244547584
memTotal: 8043147264
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.15.0-1.fc41.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.15.0
package: netavark-1.15.2-1.fc41.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.15.2
ociRuntime:
name: crun
package: crun-1.21-1.fc41.x86_64
path: /usr/bin/crun
version: |-
crun version 1.21
commit: 10269840aa07fb7e6b7e1acff6198692d8ff5c88
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20250611.g0293c6f-1.fc41.x86_64
version: |
pasta 0^20250611.g0293c6f-1.fc41.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-1.fc41.x86_64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.8.0
SLIRP_CONFIG_VERSION_MAX: 5
libseccomp: 2.5.5
swapFree: 8764166144
swapTotal: 16632504320
uptime: 1194h 35m 20.00s (Approximately 49.75 days)
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
kind-registry:5000:
Blocked: false
Insecure: true
Location: kind-registry:5000
MirrorByDigestOnly: false
Mirrors: null
Prefix: kind-registry:5000
PullFromMirror: ""
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: null
Prefix: localhost:5000
PullFromMirror: ""
search:
- kind-registry:5000
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
- quay.io
store:
configFile: /home/arsenal/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/arsenal/.local/share/containers/storage
graphRootAllocated: 168781578240
graphRootUsed: 131803070464
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /home/arsenal/.local/share/containers/storage/volumes
version:
APIVersion: 5.5.2
BuildOrigin: Fedora Project
Built: 1750723200
BuiltTime: Tue Jun 24 03:00:00 2025
GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
GoVersion: go1.23.10
Os: linux
OsArch: linux/amd64
Version: 5.5.2Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
Linux x86-64
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting