[FEAT] reconcile Domains and ClusterDomains

[skrishnan-sap]

Task List

Overall Status: Domain & ClusterDomain reconciliation implemented and working. Refactoring, unit tests in progress.

Status	Task
✅	Enhance controller by creating queues, informer change handlers etc. for Domain & ClusterDomain.
✅	Create & Update Gardener Certificate with the domain resource linked as owner via. owner identifier hash label.
✅	Implement support for changes in ingressSelector.
✅	Create & Update Gateway with domain resource in owner reference.
✅	Observe and update available subdomains from Tenants and Versions (service exposures) in CAPApplication status for use by domain resources. Requeue referenced domain resources in case of changes.
✅	Requeue referenced domain resources (added and removed) when there are changes to spec.domainRefs in the CAPApplication.
✅	Create / Update / Delete DNS Entries based on changes in DNS mode, domain host and changes queued from referencing applications.
✅	Recognize changes in domain host in domain resources to requeue referencing CAPApplications.
✅	Reque tenants and versions (with service exposure) when there is (a) a change in CAPApplication.spec.domainRefs and (b) when a domain host change ina referenced domain resource is recognized from CAPApplication. This is required to keep the VirtualServices up-to-date with the correct set of gateways and hosts.
✅	Handle deletion of domain resources including deletion of linked Certificates
✅	Refactor certificate handling to reuse common code and add support for cert-manager certificates.
✅	Rework network policies generated for ingress
❌	update unit tests.
❌	Refactor VirtualService reconciliation to reuse code for tenants and versions
❌	Implement migration routine (controller start-up)

Notes

Regarding	Note
DomainandClusterDomain`	These resources have independent reconciliation queues, but are reconciled with common functions (internal/controller/reconcile-domain.go). To create reuseable functions an abstration interface v1alpha1.DomainEntity with common getter and setter methods has been created.
Certificates	Even though it is very tempting to use the SAP Gardener provided annotations for creating certificates (TLS), it becomes harder to track the readiness of the resulting certificate for determining application readiness. Another point to note is that even though the Gardener certificates can be created by specifying an alternate namespace for the generated credentials secret (required for Istio), the certificates created with cert-manager does not support this. With these constraints the following strategy is used (a) both gardener and cert-manager certificates are created directly by the operator (b) in the namespace of the ingress controller (c) and the ready status is tracked.
Gateways	Gateways are created in the same namespace for Domains and in the cap-operator namespace for ClusterDomains. This eliminates the need to identify ingress controller namespaces during VirtualService reconciliations.
Ingress selector labels	In a cluster where multiple ingress controllers are deployed, it should be possible to switch the controller used for a domain host by changing the ingress label selector on the domain resource. This can result in a movement (delete and create) of the related certificate resource to another namespace (corresponding to new selector).
DNS Entries	DNS entries can change because of changes in (a) DNS mode (b) domain host (c) domain references in application
Network Policies	Network policies for ingress (into the cluster) are created from the domain resource. One network policy is created per namespace where the domain resource is used by an application.