The Hydra node crashes immediately when Blockfrost returns a 403 (rate limit) or the connection is reset

[aniadev]

Context & versions

• Hydra node running with Blockfrost backend
• Network: preprod
• Blockfrost client: blockfrost-haskell/0.13.0.0
• Blockfrost endpoint: cardano-preprod.blockfrost.io
• Deployment: Docker container
• Mode: online (connected to Blockfrost, not offline mode)

During normal operation, the Hydra node relies on Blockfrost APIs to query chain information.

---

Steps to reproduce

1. Configure and start a Hydra node using Blockfrost as the backend (preprod).
2. Let the node run normally and make repeated requests until the Blockfrost rate limit is reached.
3. Blockfrost starts returning HTTP 403 responses.
4. Observe the Hydra node behavior after the rate limit is hit.

Note: This can also happen if the network connection to Blockfrost is temporarily interrupted or reset.

---

Actual behavior

• When Blockfrost returns a 403 (rate limit) or the connection is reset, the Hydra node crashes immediately.
• The node does not recover automatically.
• There is no clear or user-friendly error message indicating that the crash was caused by Blockfrost rate limiting or a temporary network failure.
• From the container logs, only a low-level exception is visible:

BlockfrostAPIError "ServantClientError (ConnectionError (HttpExceptionRequest Request {
  host                 = \"cardano-preprod.blockfrost.io\"
  port                 = 443
  secure               = True
  requestHeaders       = [(\"Accept\",\"application/json;charset=utf-8,application/json\"),(\"User-Agent\",\"blockfrost-haskell/0.13.0.0\"),(\"project_id\",\"<redacted>\")]
  path                 = \"/api/v0/network/eras\"
  queryString          = \"\"
  method               = \"GET\"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}
(InternalException Network.Socket.recvBuf: resource vanished (Connection reset by peer))))"

---

Expected behavior

• The Hydra node should not crash when:

  • Blockfrost rate limits requests (HTTP 403), or
  • There is a temporary network disconnection.

• Instead, the node should:

  • Surface a clear, high-level error message explaining the Blockfrost failure.
  • Either retry with backoff, enter a degraded state, or fail gracefully.
  • Be able to recover automatically once connectivity to Blockfrost is restored.

This behavior is important because Blockfrost rate limiting or brief network interruptions are expected in real-world deployments, and a hard crash makes the node fragile and difficult to operate reliably.