supabse project goes completlty un healthy

[campuslink-web]

Our production project became unhealthy today around 09:00 AM IST.

Symptoms:

Project status is Unhealthy
Database inaccessible
Auth inaccessible
SQL Editor hangs
Table Editor hangs
Query Analyzer hangs
REST API unavailable
Flutter app offline
Web app offline
Public API offline
Restarting the project did not recover it
Another project in the same organization is healthy

This is a production outage affecting all customers.

[GaryAustin1]

Did you check your usage and database observability logs for over capacity? Disk i/o burst in particular can make a project unhealthy and restart will not work right away as it takes time to replenish. This is particularly true if you have an app doing looping error retries without backoff.

There is also infra issues that could impact restart and upgrades right now so options to resolve are more limited. See the status page.

[Jin-tonix]

Same symptoms here (started ~7/1, still ongoing 7/5). Unhealthy status, REST API timing out (10-25s) or 522/503/525. Another project in the same org (also Free tier) is healthy. Tried restarting — recovered for ~2 min then went unhealthy again. Project ref: unmbperfdjjaiqvjiijg, region ap-southeast-2.

Question re: the comment above — if repeated retries without backoff can prevent recovery via disk I/O, does that mean we should stop all requests entirely (including read-only GETs) for a few hours to let it replenish? Want to confirm before we keep polling.