Description
Today, Hypershift offers the following servicePublishingStrategies in the HostedCluster configuration.
- NodePort
- LoadBalancer
- Route
- S3
- None
And the following services can be exposed:
- APIServer
- Konnectivity
- OAuthServer
- Ignition
Our goal is to expose these services using custom Hostname and Port (e.g: 443).
As we are serving hundreds of clusters on a single cluster, we don’t want to create the hundreds of load balancers for each hostedCluster - as the “LoadBalancer/Route” strategy would create kube services with load balancer type.
For our use-case, the HostedControlPlaneOperator would create a clusterIP service on the control plane and a custom solution would be implemented to integrate with it.
Example configuration:
apiVersion: hypershift.openshift.io/v1alpha1
kind: HostedCluster
metadata:
name: clusterid
spec:
services:
- service: APIServer
servicePublishingStrategy:
type: ClusterIP
clusterIP:
hostname: clusterid.api.example.com
port: 443
- service: Konnectivity
servicePublishingStrategy:
type: ClusterIP
clusterIP:
hostname: clusterid.tunnel.example.com
port: 443
The traffic flow would look like this:
Agents/clients (DP) -----> Load Balancer listens on 443 (CP) ----> Ingress gateway (SNI based routing, TLS passthru) ----> ClusterIP service ----> Application (TLS termination)
As user services (e.g agents and clients) are running in the data plane (e.g. ha-proxy (for kube-api), konnectivity-agent (konnectivity), etc), the HostedClusterConfigOperator would configure these components with the Hostname and Port set in the servicePublishingStrategy configuration.
That's why our suggestion is to add new servicePublishingStrategy called "ClusterIP" (final name is TBD):
- it would create a ClusterIP service for the application
- on data plane, it would configure the applications with the custom Hostname and Port (e.g. konnectivity-agent)
- hostname and port would be validated. It would align with existing DNS best practices or standards (e.g., RFC 1123)
- default port would be 443
- empty hostname would be mean invalid configuration
Testing:
- the change would be backward compatible
- it would be directly additive new servicePublishingStrategy
- Unit tests
- Manual integration tests
- Would decide on further automated testing as it requires additional networking solution to test it
Alternative option: usage of special annotations on NodePort Type services
apiVersion: v1
kind: Service
metadata:
annotations:
ibm-cloud-override-service-nodeport: "443"
name: konnectivity-server
spec:
ports:
- port: 8091
protocol: TCP
targetPort: 8091
selector:
app: konnectivity-server
type: NodePort
Plese share your thoughts. Thanks!