Experiment with NGINX Ingress Controller

[jpolchlo]

AWS load balancers are surprisingly expensive—on the order of $16 per month, plus traffic/compute charges. If we consistently use LoadBalancer service types for whichever applications we wish to expose from a Kubernetes cluster, the LB charges will stack up quickly. A possible alternative may be provided by the NGINX ingress controller. It will take some investigation to determine if this can fill this need, but preliminary investigations suggest that it may be possible to use it to handle k8s Ingresses, providing a route for each ingress, with only the NGINX service requiring an AWS-provided load balancer, thereby multiplexing that LB across many applications.

This issue covers the investigation into whether this setup is possible, and also to test and document its use for future projects. If successful, subsequent issues can be created to move any deployed applications to this method.

[nanotubing]

HAProxy may be another good option here. We were evaluating it at a past job, and were impressed. we didn't end up using it, but that was for business reasons, not technical

[jpolchlo]

The content of this issue continues to be of interest to me. A noteworthy benefit to this approach would also be the ability to make simple application deployments less dependent on Terraform to set up named endpoints using route53. In the current state of things, one might deploy a Service with of LoadBalancer type, which will create an AWS ELB (in EKS). My present understanding is that this then requires creating an alias record in R53, which means that our applications will need the extra Terraform resources, which means that the application IaC is somewhat wedded to Terraform, and we can't use Kubernetes-specific tools, which may be easier or better-described (Terraform is not extremely well-documented for k8s deployments).

Using an ingress controller would imply a single alias for the cluster to be set up as part of the core infrastructure, and then applications would receive routes under some cluster-wide subdomain. For instance, what might now be jupyter.compute.azavea.com would become compute.azavea.com/jupyter, and JupyterHub would no longer require its own route53 configuration, and the jupyterhub depoyment could be performed and maintained with Helm alone—which may or may not be desirable.