Consider using `Fail` failure policy in ControlPlane's validating webhook configuration

[czeslavo]

Problem Statement

In https://github.com/Kong/gateway-operator-archive/pull/1545 we introduced provisioning of ValidatingWebhookConfiguration to the ControlPlane controller. At first attempt, we made it to use Fail policy, but it proved to highly affect tests stability. It was so because of edge cases in which the already created ValidatingWebhookConfiguration had no healthy 1) CP Deployment or 2) DP deployment (that the CP's webhook server relies on). As integration test cases are run in parallel, they were able to interfere in situations with unhealthy deployments in one of them (which is a valid use case that is being tested).

Because of the reasons above, we decided to change the failure policy to Ignore so that integration tests do not interfere with each other when their webhook service is not functional.

Proposed Solution

Change the failure policy back to Fail and ensure that ControlPlane controller is able to make sure that in case any of Deployments the admission webhook relies on (DP and CP) are not healthy, the webhook configuration is removed.

That can still be problematic in tests therfore we may be forced to keep using Ignore policy only in tests.

Additional Information

Read https://github.com/Kong/gateway-operator-archive/pull/1545#issuecomment-1980313039 for more details.

Acceptance Criteria

• Fail policy is used in ValidatingWebhookConfiguration deployed by the CP controller