Control plane and Data plane split

[strongjz]

We have worked on trying to split out the control plane and data plane with ingress-nginx. This means, running only go controller code in one deployment, running the Nginx data plane in separate pods. This leads to several advantages and design decisions we need to make.

Pros:

• Scaling - The controller and data plane have different scaling needs.
• Security - The controller needs more access than the data plane needs. This helps separate out security concerns.

Cons:

• ?

Design decisions:

• How to securely communicate the nginx config from the controller

[rikatz]

One of the major problems when implementing the split was that we had a lot of "two-way" communications:

• The dataplane pulls the configuration. But this pull, on a watch model, may fail if you have a network issue. I have been hit multiple times by gRPC watch vs experimenting to shutdown the server, and start again. So, is "pull model" the right model? (eg.: how Envoy does with xDS? I remember there is a push model). Probably some previous art research would be good
• The dataplane had to push back to control plane the events. We had this on our code. Eg.: fail to configure nginx.conf. Some other random failure
• Avoiding as much as possible the need to "write and persist" files on disk. This was one of the "most boring" problems to solve, because while we had dynamic configurations on Lua, including TLS certificates, some configuration files and some other certificates (like default certificate, the ssl ticket, dhparam) were written and read from filesystem by control plane. So there was a need to change that whole part of the code to, instead of directly writing the file on disk, we had to encode it on the communication with dataplane and make dataplane write it to disk. Can we avoid any of those writings on behalf of having NJS dynamically configuring almost everything?
• What about stop/graceful shutdown/reload? This was another feature loss.

So I think that on design decisions:

• Use NJS and dynamic configuration as much as possible (maybe the NJS endpoint can be used to CP/DP configuration, but on a different model with CP pushing the config instead)
• Define push/pull model, and caveats of each of those
• Define if we care about dataplane sending data back to Control Plane (eg events, failure to configure)
• Define if operations that may reload or shutdown the dataplane will be implemented on first step

[youngnick]

Notable here is that Envoy's model is technically a pull, in that the dataplane starts the process and the control plane sends the config in response, but then the channel is kept open (there's basically always a query from the data plane pending).

[JamesLaverack]

How to securely communicate the nginx config from the controller

I had a short chat with @strongjz offline about this, but I'll write up what I suggested to him here.

If there is an API between the Ingate control plane and the NGINX proxy fleet — without thinking about if it's push or pull, which side instigates the connection, and what format it takes — it's probably going to be some form of HTTP, either some REST structure or gRPC. Is mutual TLS a good fit here? It'd handle both encryption and authentication, and you can build the mTLS requirement into it from the start.

Then you've got a certificate management problem. cert-manager is a good fit for that (disclaimer: I used to work for Jetstack). It is extra infrastructure, but I'd speculate that many users of Ingate will be wanting to mint external TLS certificates for TLSRoutes anyway and so already were going to install it. A simple way could be to have Ingate have a "use cert-manager" mode where you configure Ingate with an Issuer, and Ingate uses the cert-manager CRDs to mint certificates for each pod and mount them into the pod filesystems. The code just needs to read a cert from disk, and hot reload it if it changes.

You could also have your API/NGINX read from K8s secrets directly and skip the mounting, but this way means that you could offer a "DIY" mode where you tell users "mount the certs in yourself, and manage them however you like". So a user could use a Vault sidecar or similar to accomplish the same — depending on their setup.

Another thing that comes to mind is SPIFFE/SPIRE. The API and NGINX pods could connect to SPIRE Workload API and get issued cert SVIDs, and use those for mTLS. You probably don't want to make SPIRE a hard dependency of Ingate, so if you did this it might need to be an optional mode.