Description
Currently, services exposed using an Ingress object can be load-balanced and are highly available because the nginx-ingress nginx services are exposed on ports 80 and 443 of every node's workload-plane interface. As such, if a client is able to failover to other server(s) (for HA), and clients connect randomly or RR to nodes in the cluster (LB), we achieve the goals.
However, many clients keep insisting a single IP address they e.g. received from DNS must work (at least until some caches get invalidated), or are unable to balance access across multiple endpoints.
As such, we need to provide virtual IPs and failover for them, as well as (potentially) load-balancing.
#779 states MetalLB needs to be deployed. Indeed, this allows to create Services of type LoadBalancer, after which a VIP will be taken from a pool of available addresses, and this VIP will at all times be ARP broadcasted by a single node in the cluster (when using L2 mode, at least: BGP changes the picture). However, this implies this one node is a bottleneck for all traffic (even though, indeed, this traffic will be propagated to one of all backing service instances to achieve LB at the backend). There's metallb/metallb#439 (comment), but even when implemented, this doesn't seem to fit nicely in the Kubernetes Service model.
This ticket is meant to discuss alternative approaches and eventually come up with a design and implementation. Ideally this is not limited to Ingress-fronted services.