Kube services considerations (#9194)

Co-authored-by: Nadine Spies <[email protected]>
Co-authored-by: soloio-bulldozer[bot] <48420018+soloio-bulldozer[bot]@users.noreply.github.com>

Author: Nadine2016

changelog/v1.17.0-beta9/docs-kube-services-note.yaml

@@ -0,0 +1,7 @@
+changelog:
+- type: NON_USER_FACING  
+  description: >- 
+    Adds considerations for selecting Kubernetes services as a routing destination instead of Upstreams. 
+    skipCI-kube-tests:true
+  issueLink: https://github.com/solo-io/gloo/issues/9164
+  resolvesIssue: true

docs/content/guides/traffic_management/destination_types/kubernetes_services/_index.md

@@ -4,17 +4,19 @@ weight: 80
 description: Routing to services registered as Kubernetes Services through the API
 ---
 
-If you are running Gloo Edge in a Kubernetes cluster, it is possible to directly specify 
-[Kubernetes Services](https://kubernetes.io/docs/concepts/services-networking/service/) as routing destinations. 
-The `kube` destination type has two required fields:
+To allow for optimal performance in Gloo Edge, it is recommended to use Gloo [static]({{% versioned_link_path fromRoot="/guides/traffic_management/destination_types/static_upstream/" %}}) and [discovered]({{% versioned_link_path fromRoot="/guides/traffic_management/destination_types/discovered_upstream/" %}}) Upstreams as your routing destination. However, if you run Gloo Edge in a Kubernetes cluster, you have the option to route to a Kubernetes service directly. 
 
-* `ref` is a {{< protobuf name="core.solo.io.ResourceRef">}} to the service that should receive traffic
-* `port` is an `int` which represents the port on which the service is listening. This must be one of the ports defined in the Kubernetes service spec
+Consider the following information before choosing a Kubernetes service as your routing destination: 
+- For Gloo Edge to route traffic to a Kubernetes service directly, Gloo Edge scans all services in the cluster and creates in-memory Upstream resources. If you have a large number of services in your cluster, the API snapshot increases which can have a negative impact on the Gloo Edge translation time.
+- When using Kubernetes services, load balancing is done in `kube-proxy` which can have further performance impacts. Routing to Gloo Upstreams bypasses `kube-proxy` as the request is routed to the pod directly. 
+- Some Gloo Edge functionality, such as policies, might not be available when using Kubernetes services. 
 
-The following configuration will forward all requests to `/petstore` to port `8080` on the Kubernetes service named 
-`petstore` in the `default` namespace.
+To use Kubernetes services as a routing destination: 
 
-{{< highlight yaml "hl_lines=6-10" >}}
+1. Enable routing to Kubernetes services by setting `settings.disableKubernetesDestinations: true` in your Gloo Edge Helm chart. By default, routing to Kubernetes services is disabled in Gloo Edge due to the negative performance impact on translation and load balancing time in clusters with a lot of Kubernetes services. 
+2. Configure the Kubernetes service as a routing destination in your VirtualService. The following example configuration forwards all requests to `/petstore` to port `8080` on the `petstore` Kubernetes service in the `default` namespace.
+
+   {{< highlight yaml "hl_lines=6-10" >}}
 routes:
 - matchers:
    - prefix: /petstore
@@ -25,4 +27,9 @@ routes:
           name: petstore
           namespace: default
         port: 8080
-{{< /highlight >}}
+   {{< /highlight >}}
+  
+   The `kube` destination type has two required fields:
+
+   * `ref` is a {{< protobuf name="core.solo.io.ResourceRef">}} to the service that receives the traffic. 
+   * `port` is an integer (`int`) and represents the port the service listens on. Note that this port must be defined in the Kubernetes service.