Deploy of ToolHive into OpenShift fails.

[dmjb]

We have received reports of ToolHive not working as an OpenShift deployment. Investigate this and make any fixes necessary.

[sabre1041]

Followed steps as described in Toolhive Operator deployment documentation. Operator deploys fine. However, when attempting to add an MCP server per the documentation, the pod (fetch-xxxxxxxx-xxxxxx) fails with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1c7c8e5]

goroutine 1 [running]:
github.com/StacklokLabs/toolhive/cmd/thv/app.checkForUpdates()
	github.com/StacklokLabs/toolhive/cmd/thv/app/commands.go:76 +0xa5
github.com/StacklokLabs/toolhive/cmd/thv/app.NewRootCmd()
	github.com/StacklokLabs/toolhive/cmd/thv/app/commands.go:57 +0x3a5
main.main()
	github.com/StacklokLabs/toolhive/cmd/thv/main.go:12 +0x13

Since OpenShift makes use of random UID/GID's at runtime, the issue is due to this requirement. By testing and relaxing the permissions to enable the pod to run with the desired UID/GID, the pod was able to start successfully. However, the actual pod that was spun up (fetch) fails with the following error:

exec /app/.venv/bin/mcp-server-fetch: permission denied

This also is due to the random UID/GID. The first pod leverages the toolhive Service Account. However, the actual fetch pod leverages the default Service Account which did not have privileges to run as a specified UID/GID.

Several enhancements are recommended to be made:

1. Support arbitrary UID/GID for the following images;
   1. ghcr.io/stackloklabs/toolhive:latest
   2. docker.io/mcp/fetch. This of course varies depending on the MCP server being deployed
2. Ability to specify the Service Account that should be used to run the MCP server

[JAORMX]

Been digging into this on the toolhive image side.

tldr; the easy reproducer is:

$ sudo podman run -u 1001010 ghcr.io/stackloklabs/toolhive:latest
Place your finger on the fingerprint reader
Trying to pull ghcr.io/stackloklabs/toolhive:latest...
Getting image source signatures
Copying blob ea18e3527952 done   | 
Copying blob 250c06f7c38e done   | 
Copying blob 5d8f20a61658 done   | 
Copying config d0eeab3725 done   | 
Writing manifest to image destination
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1c7c8e5]

goroutine 1 [running]:
github.com/StacklokLabs/toolhive/cmd/thv/app.checkForUpdates()
	github.com/StacklokLabs/toolhive/cmd/thv/app/commands.go:76 +0xa5
github.com/StacklokLabs/toolhive/cmd/thv/app.NewRootCmd()
	github.com/StacklokLabs/toolhive/cmd/thv/app/commands.go:57 +0x3a5
main.main()
	github.com/StacklokLabs/toolhive/cmd/thv/main.go:12 +0x13

(sudo because I'm running rootless podman, so I need setuid permissions)

OpenShift sets arbitrary UID/GIDs which is something we don't encounter on other setups like EKS. The image toolhive builds does not run root as default, but it tries to read a file or directory and fails there. This is probably the configuration file and clients we try to access. We should probably set flags to not read these by default. We should also allow reading the config from a configmap and set that in our deployment.

Thanks @sabre1041 for the help on the OpenShift side!

Thanks @evankanderson for the help and hints on the ko side!

[evankanderson]

Looking at the code, it feels like you might want (for container images) to package the toolhive/updates.json file (though the instance id might be something you want to generate on the fly).

Actually... thinking further, writing the updates.json back to a file on the image might be slightly tricky if you're running as a random user -- you'd need a directory with 777 permissions, or the same permissions on the file. I'm not sure what provisions ko makes for setting those permissions more open than default.

[sabre1041]

@evankanderson @JAORMX this document details some of the specific considerations needed for OpenShift

https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/images/creating-images#use-uid_create-images

[evankanderson]

Looking at the ko code, it will put the kodata files in /var/run/ko with permissions on those directories 0555 and Uid and Gid set to 0 (the default). It looks like files are also copied as 0555, with no options to customize the mode or user ID.

It looks like you can change the user embedded into the image with the --image-user flag, but you'd need to have a base image with a 777 or 775 directory to get the OpenShift magic to work on making a directory or file writeable.

[JAORMX]

I'm wondering if i can add flags in such a way that the toolhive container could just work in a read only filesystem