control plane robustness

[lgadban]

draft notes:
either block envoy updates on errors, or drop the bad route; either way propagate and log the errors
route replacement / not setting snapshot in the cache if we have errors

[lgadban]

see also:
#10506 (comment)
#10506 (comment)

[ilrudie]

Possibly helpful information for implementer. KRT has ctx.DiscardResult() which allows falling back to the previous "good" state if there is one. Take a peak at pkg/kube/krt/collection.go for details.

[timflannagan]

In addition to the issues outlined in the description, I think there's several areas in the translator that could use better error handling and/or require proper error handling instead of swallowing potential errors at lower levels.