Monitor stops after receiving "context deadline exceeded" error #254
Description
Describe the bug
When monitoring an HTTP service that receives/produces a "context deadline exceeded" error, the monitor (for that particular service) stops working entirely. No new checks are logged, neither down nor up. It does not recover when the service is back. Pausing and resuming has no effect, resetting data for that monitor does not help either. A restart is needed to get the monitor going again. EDIT: A restart does not help, but destroying the containers and deploying a new one helps temporarily, until the next error occurs.
Database
- sqlite
To Reproduce
I created a little server that reproduces the error when connecting to it. Simply run this go program and create a monitor for it on port 8080. As soon as the monitor first connects to this, the monitoring gets stuck because the programm accepts the connection, starts reading it but never responds which lets the monitor run into a timeout. (It is not the exact same error, as seen in the picture. It failed to reveive headers specifically, which my code does not, but the behaviour and base error is the same).
package main
import (
"io"
"log"
"net/http"
)
// A handler that consumes the request but never responds.
func hangHandler(w http.ResponseWriter, r *http.Request) {
// Drain the request body so the client doesn’t get a reset.
// We ignore any read error – the client may close early.
_, _ = io.Copy(io.Discard, r.Body)
_ = r.Body.Close()
log.Println("incoming request accepted, blocking response")
// Block forever – do NOT call w.WriteHeader or w.Write.
select {} // blocks indefinitely
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/", hangHandler)
srv := &http.Server{
Addr: ":8080",
Handler: mux,
}
log.Println("Hanging HTTP server listening on :8080")
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
log.Fatalf("server error: %v", err)
}
}Expected behavior
Monitoring should continue in the set intervals.
Screenshots
Notice the date, i took this today (2026-01-16)
Desktop (please complete the following information):
The error happened without accessing it from my desktop.
Smartphone (please complete the following information):
The error happened without accessing it from my phone.
Additional context
Thanks for all the work!