Extend servicePublishingStrategies in HostedCluster configuration

[mihivagyok]

Today, Hypershift offers the following servicePublishingStrategies in the HostedCluster configuration.

• NodePort
• LoadBalancer
• Route
• S3
• None

And the following services can be exposed:

• APIServer
• Konnectivity
• OAuthServer
• Ignition

Our goal is to expose these services using custom Hostname and Port (e.g: 443).
As we are serving hundreds of clusters on a single cluster, we don’t want to create the hundreds of load balancers for each hostedCluster - as the “LoadBalancer/Route” strategy would create kube services with load balancer type.

For our use-case, the HostedControlPlaneOperator would create a clusterIP service on the control plane and a custom solution would be implemented to integrate with it.

Example configuration:

apiVersion: hypershift.openshift.io/v1alpha1
kind: HostedCluster
metadata:
  name: clusterid
spec:
  services:
    - service: APIServer
      servicePublishingStrategy:
        type: ClusterIP
        clusterIP:
          hostname: clusterid.api.example.com
          port: 443
    - service: Konnectivity
      servicePublishingStrategy:
        type: ClusterIP
        clusterIP:
          hostname: clusterid.tunnel.example.com
          port: 443

The traffic flow would look like this:

Agents/clients (DP) -----> Load Balancer listens on 443 (CP) ----> Ingress gateway (SNI based routing, TLS passthru) ----> ClusterIP service ----> Application (TLS termination)

As user services (e.g agents and clients) are running in the data plane (e.g. ha-proxy (for kube-api), konnectivity-agent (konnectivity), etc), the HostedClusterConfigOperator would configure these components with the Hostname and Port set in the servicePublishingStrategy configuration.

That's why our suggestion is to add new servicePublishingStrategy called "ClusterIP" (final name is TBD):

• it would create a ClusterIP service for the application
• on data plane, it would configure the applications with the custom Hostname and Port (e.g. konnectivity-agent)
• hostname and port would be validated. It would align with existing DNS best practices or standards (e.g., RFC 1123)
• default port would be 443
• empty hostname would be mean invalid configuration

Testing:

• the change would be backward compatible
• it would be directly additive new servicePublishingStrategy
• Unit tests
• Manual integration tests
• Would decide on further automated testing as it requires additional networking solution to test it

Alternative option: usage of special annotations on NodePort Type services

apiVersion: v1
kind: Service
metadata:
  annotations:
    ibm-cloud-override-service-nodeport: "443"
  name: konnectivity-server
spec:
  ports:
  - port: 8091
    protocol: TCP
    targetPort: 8091
  selector:
    app: konnectivity-server
  type: NodePort

Plese share your thoughts. Thanks!