Expand alert coverage

[travjenkins]

Potential alerts

If task is down (shard failures)

• The shard failed more than x times in n minutes
• Do not have triggers today on shard failures. This would need to be added.
• Want to filter out inferred schema updates

Your schema has changed / going to backfill (incompatible schema)

• Should be on materialization when it responds to an incompatible schema change
• We get a lot of schema updates that do not cause any issues

Publication Failure

• If there was any failures in automation for autoDiscover pubs or sourceCaptures pubs

Much more details to come:

https://docs.google.com/document/d/1XOxMlug7OtOAogerSnb-j4-dR9xfk6mImNtPHoujFVo/edit