implement REST API

[garlick]

Problem: a REST API to Flux would allow users to leverage significant web tooling in their applications/workflow orchestrators/whatever.

There is a specific need at LLNL to enable access via a proxy server called wormhole.

See also

• Early issue: Create a "REST API router" flux-core#2131
• Prototype for cloud convergence: https://github.com/flux-framework/flux-restful-api
• Wormhole presentation: https://hpc.llnl.gov/sites/default/files/2025-10/2025-10-14-wormhole.pdf
• Slurm REST details: https://slurm.schedmd.com/rest.html

[garlick]

@tpatki feel free to add requirements here.

[garlick]

Here's a point design and proof of concept for a socket-activated, per-user REST server that delegates the enterprise security interface to an nginx reverse proxy. This is the product of some collaborative design with Claude. The server is a standalone process that translates REST requests on a UNIX domain socket to Flux requests to Flux instances that the user has access to. In the production deployment, the proxy would probably be deployed on a login node, and at any given time there would be one REST server per active REST user (not per job/instance). The hope is that the standalone process, without the baggage of multi-user support or web security, only needs to solve the protocol translation problem.

https://github.com/garlick/flux-rest-server

I did test the socket activation on my test cluster (instructions are in the README). I didn't yet try configuring the nginx proxy.

To be clear, the server component here is just a throw away demo. There are some sharness tests that check that it offers the correct interface for integration into the socket activation strategy to help make replacing it a little easier.

[grondo]

Nice! I haven't done any due diligence researching this topic, so my opinion should be weighted accordingly, but what you have here matches closely what I was picturing as an initial design! Nice work putting this together.

[garlick]

Glad to hear we are aligned. I started out with a server embedded in the broker as I had originally proposed, and that led to extra "plumbing" problems that went away with the approach closer to what you intuited was a good idea all along!

[tpatki]

Hi @garlick @grondo: Thank you for putting this together, this is a great starting point (what I called the "lazy" multi-user) and I was looking into something similar with Claude using FASTAPI/JWT support for a per-user basis. This approach may work for a few select users that need this functionality for test purposes (I haven't looked at your code yet...will test it out shortly).

However, my understanding here is that while the per-user use case is helpful to test out how to add JWT support, it may not be helpful from the perspective of supporting the model that Brian was thinking of. Also, it seems infeasible to run a persistent process for each user in production on a login node (and, if the user has a Flux allocation already, they can submit jobs with flux submit and don't really need a REST API.), but we can worry about that later.

The use case that I discussed with Brian and @trws was being able to have a persistent flux-restd style daemon, that authenticates an incoming user's JWT token, and then translates the credentials to be able to submit a job as that user. @trws seemed to propose another idea that leveraged UIDs and munge but it wasn't clear to me -- we can discuss once he's back from ISC.

[garlick]

Hi @tpatki - to clarify, this proposal does not run a persistent process for each user on the login node. They are spawned on demand (the "socket activated" part) when a user gets a request past the nginx server, and then terminates after an idle period. The nginx front end takes care of the security and is what users interact directly with.

The flux-rest-server running as the user translates HTTP requests to Flux RPCs and since it is running as the user can sign job requests on behalf of the user who was authenticated by nginx.

The nice thing IMHO is the separation of concerns: the flux component can assume authentication and user transitions are already handled by components that are hardened and well understood by admins.

However, my understanding here is that while the per-user use case is helpful to test out how to add JWT support, it may not be helpful from the perspective of supporting the model that Brian was thinking of

To be clear, this is a multi-user proposal. The per-user service starts out connected to the system instance and is ready to interact with Flux at that level (e.g. submit jobs to the system not to a user instance). It could then "browse" to sub-instances if desired.

[grondo]

Also, I could be mistaken, but I think from the external REST client perspective, this would look identical to a persistent flux-restd implementation (which I assume would have to run as the flux user or root. Possibly root in order to sign user jobspec for submit).

[tpatki]

Hi @garlick @grondo: Thank you for clarifying this, I understand the design now. I've been looking at flux-rest-server as a starting point. The per-user socket-activation design makes sense — server runs as the user's UID, OS vouches for identity via the domain socket (let me know if misunderstood this).

The nice thing IMHO is the separation of concerns: the flux component can assume authentication and user transitions are already handled by components that are hardened and well understood by admins.

My question is about external clients, such as those using bearer tokens (eg with JWT). For example, the LC Wormhole bearer token that is valid for 45 days (see John's slide deck, slide 2), or something like an external CI/CD pipeline that could be storing the token as a secret and triggering Flux jobs, or a interactive JupyterHub notebook.

My question is about the division of responsibility for authentication: who should validate the bearer token (eg in the LC wormhole case) and how would they persist? It seems like with the nginx approach, we'd need the LC wormhole to provide a trusted user identity each time to pass along to Flux, so the flux components can assume existing authentication. I'm not sure how this would come together just yet... thoughts?

[garlick]

In this proposal nginx is the front end that would interact with wormhole. If wormhole wants a bearer token for fast reuse of connections, I think nginx can do that, although my assumption when I looked at that diagram was that those bearer tokens were issued by wormhole and would not pass through to the eventual service on the cluster. Wormhole's proxy connection to the services would be a separately authenticated and secured thing?

In any case, nginx would take care of what wormhole or direct users need, whether that be OAuth+JWT or whatever, and flux-rest-server just answers bare HTTP requests on a unix domain socket with the assumption that what's on the other end is nginx acting solely on behalf of the user that triggered the socket activation.

Caveat: I have no direct experience working with nginx or web tooling in general, so I may be out of my depth!

p.s. https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-jwt-authentication/

[garlick]

Just tested with the nginx front end (insecure config example) on my test cluster and found an error in the nginx config. I pushed a fixup in case anyone else wants to play.

garlick@picl0:~$   curl -u "$USER" http://localhost:8080/api/v1/
Enter host password for user 'garlick':
{"name": "flux-rest-server", "user": "garlick", "broker_version": "0.86.0-63-gb2e84a776", "rank": 0, "size": 8}

[garlick]

The security architecture seemed to be an important thing to resolve first in this design, but another important topic raised by @trws is creating a "machinable" API. That is, make the REST protocol discoverable such that APIs can be automatically generated. See also

• https://www.openapis.org/
• https://slurm.schedmd.com/openapi_release_notes.html