Prow: Alerting #930
Description
Split out from #896
When #896 is done, we will have the basics in place that will allow us to define alerts.
This issue is for doing that.
Define some basic alerts (or use those that comes with kube-prometheus), set up receivers for slack and/or email.
Useful alerts would be for example
- generic kubernetes issues like unhealthy nodes (see this)
- unhealthy workloads (except tests, they can be expected to fail), for example, prow,