Add global down detection

[ericswpark]

If more than n number of webring members' sites go down, treat it as an upstream outage and not an individual website failure.

[kdkasad]

I don't think we should do this. Not all sites are hosted using the same provider, so it doesn't make sense to say "everything is down" if it's just all Cloudflare-hosted sites that are down.

The outage earlier today seems to have affected 4 out of 21 sites, which doesn't feel like a particularly large portion.

Also, we do have a mechanism to detect if the webring's internet connection is down, and if so, it won't alert about not being able to reach the sites in the ring.

[ericswpark]

@kdkasad I was thinking if three or more go down at once, it still sends a message but makes silent pings since it's not urgent. Example message:

Warning: 3 sites from the webring went offline at once. There may be a provider outage.
@/ericswpark your site returned an error response: 500 (Internal Server Error)
@/another-username your site returned an error response: 500 (Internal Server Error)
@/another-username your site returned an error response: 500 (Internal Server Error)

And possibly more heuristics (matching against 5xx only?)

[9p4]

What about keeping state so that notifications are only sent once.

[kdkasad]

What about keeping state so that notifications are only sent once.

This is what we do. We only send a notification once per outage per 24 hours. The repeated notifications are because the sites are coming back online, then going down again, which gets marked as a new outage.

I think we should just remove the "once per outage" part and make the notifications once per 24 hours, no matter how many times the site goes up/down within that period.

Does this work for you @ericswpark?

It's easier to implement than the global outage detection, as we currently don't collect the results of all the site checks; we perform them concurrently and dispatch notifications concurrently as well.

[ericswpark]

I would still like to try to attempt it, but feel free to close if you think the development time is not worth it.

[kdkasad]

Ok, go ahead then. Let's make the "certain number" part of the DiscordTable so it can be controlled by config.toml