kube-vip should declare hostPort to avoid port scheduling conflicts

[rptaylor]

When addressing #82 I relied on the fact that kube-vip evidently always serves prometheus metrics on host port 2112 no matter what ( which it is able to do by virtue of having hostNetwork).

In doing so, kube-vip is actually a bit sneaky/mischievous, because it occupies host port 2112 without telling k8s, so the k8s scheduler does not have a way to avoid host port conflicts. Maybe that is a somewhat moot point as a daemonset usually runs everywhere (modulo tolerations), but strictly speaking it should declare a hostPort. At least, this way if something else on the cluster gets deployed with hostPort 2112, k8s will prevent it, instead of the apps encountering run-time crashes/conflicts.

This means that the containerPort defined in the daemonset https://github.com/kube-vip/helm-charts/blob/main/charts/kube-vip/templates/daemonset.yaml#L30
should , the way things are currently, always be present unconditionally, and also with hostPort included along with it.

However thinking about it a bit more, it might be preferable to have a flag to disable kube-vip from serving metrics on :2112
@thebsdbox is there a way to do that?

[rptaylor]

However, it's worth noting that declaring a hostPort, while arguably a good practice to avoid port conflicts, would also constitute an additional security elevation as far as tools like Kyverno are concerned. All ports used by a hostNetwork are implicitly functionally equivalent to hostPorts (although k8s might not even know about them) so there is no real difference security-wise in practical terms, but Kyverno will nevertheless block the pods unless there is a PolicyException allowing use of hostPorts, in addition to hostNetwork.

That's another reason it would probably be good to have a flag to control whether kube-vip uses a port for metrics.