RFC: Allow running the agent container without additional capabilities through a separate binary

[nadiamoe]

Note

This issue is a RFC, which means it lays out a problem, a potential solution, and requests comments on it.

Problem

There's a use case for running the synthetic monitoring agent on a container on environments that do not allow granting additional capabilities to the container. This may happen, for example, by enforcing the Baseline pod security standard.

Currently, this poses a problem, as attempting to run the synthetic-monitoring-agent container without the NET_RAW capability will prevent the container from starting. The reason for this is that the binary that acts as the container entrypoint has this capability added:

synthetic-monitoring-agent/Dockerfile

Line 15 in b7587fa

RUN setcap cap_net_raw=+ep /usr/local/bin/synthetic-monitoring-agent

As a security precaution, the linux kernel will refuse to run a binary requesting a capability that is not part of the permitted set, and it results on an error like the following:

sh: /usr/local/bin/synthetic-monitoring-agent: Operation not permitted

The reason for the synthetic-monitoring-agent binary to request NET_RAW is to be able to perform ICMP-based checks, such as ping or traceroute, which require forging raw ICMP packets. While there's a way to perform a subset of these checks using UDP packets, these do not have complete feature parity with ICMP.

Proposed solution

I suggest that we add a second binary to the same container image, which is a build-time copy of the original binary, but without the NET_RAW capability added. This is a Dockerfile-only change, which would look like this:

diff --git a/Dockerfile b/Dockerfile
index 93f8b46..2ad16f6 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -25,6 +25,7 @@ RUN adduser -D -u 12345 -g 12345 sm
 
 ADD --chown=sm:sm --chmod=0500 https://github.com/grafana/xk6-sm/releases/download/v0.0.3-pre/sm-k6-${TARGETOS}-${TARGETARCH} /usr/local/bin/sm-k6
 COPY --chown=sm:sm --chmod=0500 --from=setcapper /usr/local/bin/synthetic-monitoring-agent /usr/local/bin/synthetic-monitoring-agent
+COPY --chown=sm:sm --chmod=0500 dist/${HOST_DIST}/synthetic-monitoring-agent /usr/local/bin/synthetic-monitoring-agent-unprivileged
 COPY --chown=sm:sm scripts/pre-stop.sh /usr/local/lib/synthetic-monitoring-agent/pre-stop.sh
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

With the difference being that synthetic-monitoring-agent-unprivileged is not granted any file-based capability, unlike synthetic-monitoring-agent which is:

synthetic-monitoring-agent/Dockerfile

Line 15 in b7587fa

RUN setcap cap_net_raw=+ep /usr/local/bin/synthetic-monitoring-agent

This second binary allows users who do not want or can add the NET_RAW capability to the container to still run the container, acknowledging that functionality related to ICMP checks may not work correctly. They would be able to do that by overriding the default entrypoint, which is something universally supported by container runtimes. We would document this in the repo readme and in the grafana docs website.

• ➕ Pros:
  • Low maintenance cost: This second binary is identical to the main one, it does not need to be built separately.
• ➖ Cons:
  • The size of the container image will increase

Alternative solutions

1. Do nothing, the use case remains unsupported.
2. Same "two binaries" approach as above, but making unprivileged the default.
3. Creating an -unprivileged container flavor.
4. Granting the NET_RAW capability in runtime.

[peterschretlen]

Related issue: #1251

[mem]

Something is off here (or at least the description is incomplete and it's assuming something).

Without a corresponding command line, this output is meaningless:

sh: /usr/local/bin/synthetic-monitoring-agent: Operation not permitted

Without extra flags, synthetic-monitoring-agent won't start, so something has been left out here.

I cannot reproduce that behavior locally, so something is missing:

$ filecap $PWD/sm-agent/dist/synthetic-monitoring-agent

$ # ^ no extra capabilities

$ ls -ln ./sm-agent/dist/synthetic-monitoring-agent
-rwxr-xr-x 1 1000 1000 29957553 Mar 31 09:40 ./sm-agent/dist/synthetic-monitoring-agent

$ # ^ not setuid

$ id -u
1000

$ # ^ not root

$ ./sm-agent/dist/synthetic-monitoring-agent -api-token "$(cat marcelo-agent-smdev.dat)" -api-server-address synthetic-monitoring-grpc-dev.grafana-dev.net:443 -k6-uri $PWD/xk6-sm/dist/k6 -features k6 
{"program":"synthetic-monitoring-agent","subsystem":"http","time":1743436414881,"message":"Starting HTTP server on localhost:4050..."}
2025/03/31 09:53:35 RequireTransportSecurity

at this point the agent is running.

These are the corresponding results:

As you can see, the check runs and obtains the corresponding data.

This has been in place for quite a long time. I believe the only reason why we run ping checks with additional capabilities in production is because we want to use ICMP (which requires raw sockets) instead of UDP (which does require additional set up in the host system). There's code in the prober to use UDP if available and it falls back to ICMP if not.

[peterschretlen]

If I create an agent container using the browser version of the image + dropping the NET_RAW capability, I get this error:

❯ docker run --cap-drop=NET_RAW grafana/synthetic-monitoring-agent:v0.34.1-browser --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --debug=true
[FATAL tini (7)] exec /usr/local/bin/synthetic-monitoring-agent failed: Operation not permitted

Same if I use the image without browser:

❯ docker run --cap-drop=NET_RAW grafana/synthetic-monitoring-agent:v0.34.1 --api-server-address=${API_SERVER} --api-token=${API_TOKEN} --debug=true
exec /usr/local/bin/synthetic-monitoring-agent: operation not permitted

[peterschretlen]

My concern is having to support/maintain/verify/be-aware-of the privileged & unprivileged modes of operating. It's another variable in private probes which may be warranted, but it make it harder to troubleshoot and support (I can anticipate the "why don't my ping checks work on this private probe?" question).

[nadiamoe]

Let me elaborate the capabilities problem a bit more:

If the binary does not have any capabilities added, your container runtime ✅ will run it. If you are not root (as it is now set in the Dockerfile), traceroute checks and setting DF flags will fail. This is well known.

If the binary does have the capabilities added (as it has now), the situation gets more complex:

In docker, it ✅ will run by default. That's because docker has a default set of permitted capabilities, which includes NET_RAW. If you override that default with --cap-drop=NET_RAW (or --cap-drop=all), then the binary ❌ will not start (regardless of arguments), as the kernel is forbidding it from doing so.

In Kubernetes, whether it will run with a default securityContext will depend on your CRI, which has the equivalent of the default capabilities I linked above for docker. On CRI-O, it will ❌ not run with an empty (default) capabilities section. This is the reason why our example adds this capability:

synthetic-monitoring-agent/examples/kubernetes/deployment.yaml

Lines 47 to 51 in 2699f62

	capabilities:
	add:
	- NET_RAW # Needed for ICMP (ping and traceroute) checks.
	drop:
	- all

In Kubernetes, the container ✅ will run with this configuration, and would also run without the drop part, which we include on the example simply because it's good practice.

In your local computer, ✅ will run by default. That's because outside of containers, linux has a very wide list of capabilities allowed for non-root users. For example, on my workstation:

12:34:33 /tmp/nadia $> capsh --print
Current: cap_wake_alarm=i
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
Ambient set =
Current IAB: cap_wake_alarm
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=1000(nadia) euid=1000(nadia)
gid=100(users)
groups=10(wheel),14(uucp),56(bumblebee),100(users),108(vboxusers),150(wireshark),985(adbusers),987(boinc),991(libvirt),992(plugdev),994(docker)
Guessed mode: HYBRID (4)

If you squint you can see net_raw in there.

So, summarizing: Container runtimes change the permitted set, which will cause the kernel to refuse to execute any binary declaring any capability not present on it. Different runtimes have different defaults, with Docker's being particularly permissive. Hopefully this helped!