Skip to content

fix: run MONITOR on a dedicated connection and propagate read errors - #3901

Open
ankit-songara wants to merge 6 commits into
redis:masterfrom
ankit-songara:fix-monitor-readtimeout-hang
Open

fix: run MONITOR on a dedicated connection and propagate read errors#3901
ankit-songara wants to merge 6 commits into
redis:masterfrom
ankit-songara:fix-monitor-readtimeout-hang

Conversation

@ankit-songara

@ankit-songara ankit-songara commented Jul 13, 2026

Copy link
Copy Markdown

Fixes #3079

Problem

Two layered bugs, the second discovered while CI's race detector ran the regression test for the first:

1. Read errors were silently swallowed (the reported issue). readMonitor discards the error from rd.Peek(1), so once the connection died the goroutine spun in a hot loop forever re-peeking it, cmd.Err() stayed nil, and a listener blocked on the monitor channel hung indefinitely with no way to detect the failure.

2. MONITOR ran on a pooled connection. MonitorCmd.readReply spawns the reader goroutine and returns immediately, so withConn put the connection back into the pool while the goroutine was still reading from it. The pool's putConn health-check peek then read from the same bufio.Reader concurrently — a data race (CI flagged it on the first version of this PR). It also meant the client's ReadTimeout stayed armed on the connection, which is why a monitor on a quiet server broke after a few seconds in the first place.

Changes

  • _process routes MonitorCmd to a new processMonitor, which runs the command on a dedicated, non-pooled connection (ConnPool.NewConn) with no read deadline — matching both the doc comment's existing promise ("It needs a dedicated connection") and how MONITOR behaves in redis-cli. An idle server no longer breaks the monitor.
  • On a read error the channel is closed and the error is available via cmd.Err(), so a blocked listener wakes up (msg, ok := <-ch with ok == false) — option 2 from the issue.
  • Stop() closes the dedicated connection to unblock a reader waiting in Peek for traffic. A clean stop reports no error and leaves the channel open (unchanged from before).
  • readMonitor no longer holds the mutex across the blocking Peek (that would deadlock Stop()); the mutex now only guards the status field. The reader goroutine is the sole reader of the connection, so no lock is needed around reads.

Testing

Two regression tests using a fake in-process TCP server speaking just enough RESP for MONITOR — they need no real Redis and are not gated behind RUN_MONITOR_TEST:

  • TestMonitorConnErrorClosesChannel: server drops the connection mid-monitor → channel closes, Err() is set. Hangs forever on master.
  • TestMonitorStopWithoutTraffic: Stop() takes effect with zero server traffic and reports no error. Deadlocks or times out on master since the reader only observed the stop flag when data arrived.

Note

Medium Risk
Changes core command dispatch and long-lived connection handling for MONITOR; behavior is more correct but affects anyone relying on the old pooled-connection semantics.

Overview
Fixes MONITOR lifecycle bugs (#3079): pooled connections, silent read failures, and Stop blocking when the server is idle.

MONITOR now runs on a dedicated non-pooled connection via processMonitor (NewConn, no read deadline), so idle servers do not hit ReadTimeout and the background reader no longer races with pool health checks on the same bufio.Reader.

On connection failure, readMonitor surfaces Peek/ReadString errors, sets cmd.Err(), and closes the monitor channel so blocked consumers can exit. A clean Stop() closes that connection to unblock Peek, leaves the channel open, and does not set an error; readMonitor avoids holding the mutex across blocking reads and treats stop-induced close as success.

Docs note the new behavior; CloseReasonMonitor labels dedicated monitor teardown. Regression tests use a fake TCP server for connection drop and stop-without-traffic.

Reviewed by Cursor Bugbot for commit 4176bec. Bugbot is set up for automated code reviews on this repo. Configure here.

…ever

When the connection backing a MONITOR command dies (most commonly the
client's ReadTimeout expiring because no traffic arrived), the read error
was silently discarded: rd.Peek's error was ignored in readMonitor, so the
goroutine span forever re-peeking a dead connection, cmd.Err() stayed nil,
and a listener blocked on the channel hung indefinitely.

Now readMonitor returns the peek error, and readReply cancels the context
and closes the channel, so a blocked listener wakes up and can inspect
cmd.Err().

Adds a regression test using a fake in-process server that streams one
monitor line and then goes silent; the test fails on the old code and
passes now. Also documents the channel-close behavior on Monitor.

Fixes redis#3079
MONITOR previously executed like a regular command: readReply spawned the
reader goroutine and returned, so the connection went straight back into
the pool while the goroutine kept reading from it. The pool's health check
(and any command that got the connection next) then read from the same
bufio.Reader concurrently - a data race. On top of that, the client's
ReadTimeout stayed armed, so a monitor on a quiet server died after a few
seconds, and the resulting error was silently discarded: rd.Peek's error
was ignored, the goroutine span forever re-peeking a dead connection,
cmd.Err() stayed nil, and a listener blocked on the channel hung forever.

The docs already say MONITOR "needs a dedicated connection"; now it gets
one. _process routes MonitorCmd to a non-pooled connection with no read
deadline, matching how the command behaves in redis-cli. On a read error
the channel is closed and the error is available via cmd.Err(), so
listeners are not blocked forever. Stop() closes the dedicated connection
to unblock a reader waiting for traffic and shuts down cleanly without
reporting an error.

Adds regression tests using a fake in-process server, so they run without
a real Redis and are not gated behind RUN_MONITOR_TEST.

Fixes redis#3079
@ankit-songara ankit-songara changed the title fix: close monitor channel on read error so listeners don't block forever fix: run MONITOR on a dedicated connection and propagate read errors Jul 13, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 53941c2. Configure here.

Comment thread command.go
Comment thread redis.go Outdated
- Treat a read error during ReadString as a clean shutdown when Stop
  closed the connection, matching the Peek path; previously a Stop that
  raced with an in-flight line reported a spurious error and closed the
  channel.
- Tear down the dedicated monitor connection via connPool.CloseConn
  instead of a bare cn.Close, so the pool's connection map and metrics
  stay consistent; adds pool.CloseReasonMonitor.

@ndyakov ndyakov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this contribution @ankit-songara. I did close the previous fix which was not ideal anyway and I do prefer this one. Will review this one as soon as possible.

@ankit-songara
ankit-songara requested a review from ndyakov July 16, 2026 19:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

monitor command breaks due to ReadTimeout

3 participants