exec container process `/gatus`: Operation not permitted

[edvgui]

Describe the bug

Podman rootless can not start a container using the latest image.

What do you see?

$ podman run --rm ghcr.io/twin/gatus:latest
Trying to pull ghcr.io/twin/gatus:latest...
Getting image source signatures
Copying blob d4c216b153d7 done   | 
Copying blob 92283a3809e7 done   | 
Copying blob 10ea3898eea5 done   | 
Copying config c7c2437507 done   | 
Writing manifest to image destination
{"msg":"exec container process `/gatus`: Operation not permitted","level":"error","time":"2025-04-02T06:06:08.274217Z"}

What do you expect to see?

$ podman run --rm ghcr.io/twin/gatus:v5.17.0
Trying to pull ghcr.io/twin/gatus:v5.17.0...
Getting image source signatures
Copying blob 838a07d22073 done   | 
Copying blob 396d5518efcc done   | 
Copying blob cf2d56fcd7b1 done   | 
Copying config 7286de66b3 done   | 
Writing manifest to image destination
2025/04/02 06:13:14 [main.configureLogging] Log Level is set to INFO
2025/04/02 06:13:14 [config.LoadConfiguration] Reading configuration from configFile=config/config.yaml
2025/04/02 06:13:14 [config.validateAlertingConfig] Alerting is not configured
2025/04/02 06:13:14 [config.validateEndpointsConfig] Validated 7 endpoints
2025/04/02 06:13:14 [config.validateEndpointsConfig] Validated 0 external endpoints
2025/04/02 06:13:14 [store.Initialize] Creating storage provider of type=memory
2025/04/02 06:13:14 [controller.Handle] Listening on 0.0.0.0:8080
...

List the steps that must be taken to reproduce this issue

1. Get podman
2. Run the container using podman and latest image (rootless)

Version

latest dev (8a62eb0)

Additional information

Turns out that adding the CAP_NET_RAW capability to the container fixes the issue:

podman run --rm --cap-add cap_net_raw ghcr.io/twin/gatus:latest

It would be nice to add the capability to the image directly, but that didn't seem to work for me either:

$ cat Dockerfile 
from ghcr.io/twin/gatus:latest
LABEL io.containers.capabilities=cap_net_raw

$ podman build -t gatus:test .
STEP 1/2: FROM ghcr.io/twin/gatus:latest
STEP 2/2: LABEL io.containers.capabilities=cap_net_raw
COMMIT gatus:test
--> 974b470e7a8e
Successfully tagged localhost/gatus:test
974b470e7a8e2bc4898d17602af9051d27fd25f04a04c5b10192aaa796b83ab3
$ podman run --rm gatus:test
ERRO[0000] Capabilities requested by user or image are not allowed by default: "CAP_NET_RAW" 
{"msg":"exec container process `/gatus`: Operation not permitted","level":"error","time":"2025-04-02T06:28:31.273567Z"}

I am not sure what is the right solution here, but I do find it a bit annoying that this capability is suddenly to be manually added to any gatus container, even if gatus doesn't need to perform icmp checks.

[TwiN]

Hmm, I tested #1056 with Docker, but not with Podman.

[jamesread]

Reproducible on Kubernetes/crio with a default (empty), too. Previous images worked, but I didn't use Gatus in a while so I cannot isolate which image version caused this to break.

This is likely because Docker runs containers as root by default, whereas podman, crio and other CRIs discourage root.