Skip to content

Monitor stops after receiving "context deadline exceeded" error #254

@Pandabrain

Description

@Pandabrain

Describe the bug
When monitoring an HTTP service that receives/produces a "context deadline exceeded" error, the monitor (for that particular service) stops working entirely. No new checks are logged, neither down nor up. It does not recover when the service is back. Pausing and resuming has no effect, resetting data for that monitor does not help either. A restart is needed to get the monitor going again. EDIT: A restart does not help, but destroying the containers and deploying a new one helps temporarily, until the next error occurs.

Database

  • sqlite

To Reproduce
I created a little server that reproduces the error when connecting to it. Simply run this go program and create a monitor for it on port 8080. As soon as the monitor first connects to this, the monitoring gets stuck because the programm accepts the connection, starts reading it but never responds which lets the monitor run into a timeout. (It is not the exact same error, as seen in the picture. It failed to reveive headers specifically, which my code does not, but the behaviour and base error is the same).

package main

import (
	"io"
	"log"
	"net/http"
)

// A handler that consumes the request but never responds.
func hangHandler(w http.ResponseWriter, r *http.Request) {
	// Drain the request body so the client doesn’t get a reset.
	// We ignore any read error – the client may close early.
	_, _ = io.Copy(io.Discard, r.Body)
	_ = r.Body.Close()

	log.Println("incoming request accepted, blocking response")

	// Block forever – do NOT call w.WriteHeader or w.Write.
	select {} // blocks indefinitely
}

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/", hangHandler)

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	log.Println("Hanging HTTP server listening on :8080")
	if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
		log.Fatalf("server error: %v", err)
	}
}

Expected behavior
Monitoring should continue in the set intervals.

Screenshots
Notice the date, i took this today (2026-01-16)
Image

Desktop (please complete the following information):
The error happened without accessing it from my desktop.

Smartphone (please complete the following information):
The error happened without accessing it from my phone.

Additional context
Thanks for all the work!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions