fix: run MONITOR on a dedicated connection and propagate read errors - #3901
Open
ankit-songara wants to merge 6 commits into
Open
fix: run MONITOR on a dedicated connection and propagate read errors#3901ankit-songara wants to merge 6 commits into
ankit-songara wants to merge 6 commits into
Conversation
…ever When the connection backing a MONITOR command dies (most commonly the client's ReadTimeout expiring because no traffic arrived), the read error was silently discarded: rd.Peek's error was ignored in readMonitor, so the goroutine span forever re-peeking a dead connection, cmd.Err() stayed nil, and a listener blocked on the channel hung indefinitely. Now readMonitor returns the peek error, and readReply cancels the context and closes the channel, so a blocked listener wakes up and can inspect cmd.Err(). Adds a regression test using a fake in-process server that streams one monitor line and then goes silent; the test fails on the old code and passes now. Also documents the channel-close behavior on Monitor. Fixes redis#3079
MONITOR previously executed like a regular command: readReply spawned the reader goroutine and returned, so the connection went straight back into the pool while the goroutine kept reading from it. The pool's health check (and any command that got the connection next) then read from the same bufio.Reader concurrently - a data race. On top of that, the client's ReadTimeout stayed armed, so a monitor on a quiet server died after a few seconds, and the resulting error was silently discarded: rd.Peek's error was ignored, the goroutine span forever re-peeking a dead connection, cmd.Err() stayed nil, and a listener blocked on the channel hung forever. The docs already say MONITOR "needs a dedicated connection"; now it gets one. _process routes MonitorCmd to a non-pooled connection with no read deadline, matching how the command behaves in redis-cli. On a read error the channel is closed and the error is available via cmd.Err(), so listeners are not blocked forever. Stop() closes the dedicated connection to unblock a reader waiting for traffic and shuts down cleanly without reporting an error. Adds regression tests using a fake in-process server, so they run without a real Redis and are not gated behind RUN_MONITOR_TEST. Fixes redis#3079
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
Reviewed by Cursor Bugbot for commit 53941c2. Configure here.
- Treat a read error during ReadString as a clean shutdown when Stop closed the connection, matching the Peek path; previously a Stop that raced with an in-flight line reported a spurious error and closed the channel. - Tear down the dedicated monitor connection via connPool.CloseConn instead of a bare cn.Close, so the pool's connection map and metrics stay consistent; adds pool.CloseReasonMonitor.
ndyakov
reviewed
Jul 16, 2026
Member
There was a problem hiding this comment.
Thank you for this contribution @ankit-songara. I did close the previous fix which was not ideal anyway and I do prefer this one. Will review this one as soon as possible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Fixes #3079
Problem
Two layered bugs, the second discovered while CI's race detector ran the regression test for the first:
1. Read errors were silently swallowed (the reported issue).
readMonitordiscards the error fromrd.Peek(1), so once the connection died the goroutine spun in a hot loop forever re-peeking it,cmd.Err()stayed nil, and a listener blocked on the monitor channel hung indefinitely with no way to detect the failure.2. MONITOR ran on a pooled connection.
MonitorCmd.readReplyspawns the reader goroutine and returns immediately, sowithConnput the connection back into the pool while the goroutine was still reading from it. The pool'sputConnhealth-check peek then read from the samebufio.Readerconcurrently — a data race (CI flagged it on the first version of this PR). It also meant the client'sReadTimeoutstayed armed on the connection, which is why a monitor on a quiet server broke after a few seconds in the first place.Changes
_processroutesMonitorCmdto a newprocessMonitor, which runs the command on a dedicated, non-pooled connection (ConnPool.NewConn) with no read deadline — matching both the doc comment's existing promise ("It needs a dedicated connection") and how MONITOR behaves in redis-cli. An idle server no longer breaks the monitor.cmd.Err(), so a blocked listener wakes up (msg, ok := <-chwithok == false) — option 2 from the issue.Stop()closes the dedicated connection to unblock a reader waiting inPeekfor traffic. A clean stop reports no error and leaves the channel open (unchanged from before).readMonitorno longer holds the mutex across the blockingPeek(that would deadlockStop()); the mutex now only guards the status field. The reader goroutine is the sole reader of the connection, so no lock is needed around reads.Testing
Two regression tests using a fake in-process TCP server speaking just enough RESP for MONITOR — they need no real Redis and are not gated behind
RUN_MONITOR_TEST:TestMonitorConnErrorClosesChannel: server drops the connection mid-monitor → channel closes,Err()is set. Hangs forever on master.TestMonitorStopWithoutTraffic:Stop()takes effect with zero server traffic and reports no error. Deadlocks or times out on master since the reader only observed the stop flag when data arrived.Note
Medium Risk
Changes core command dispatch and long-lived connection handling for MONITOR; behavior is more correct but affects anyone relying on the old pooled-connection semantics.
Overview
Fixes MONITOR lifecycle bugs (#3079): pooled connections, silent read failures, and Stop blocking when the server is idle.
MONITOR now runs on a dedicated non-pooled connection via
processMonitor(NewConn, no read deadline), so idle servers do not hitReadTimeoutand the background reader no longer races with pool health checks on the samebufio.Reader.On connection failure,
readMonitorsurfacesPeek/ReadStringerrors, setscmd.Err(), and closes the monitor channel so blocked consumers can exit. A cleanStop()closes that connection to unblockPeek, leaves the channel open, and does not set an error;readMonitoravoids holding the mutex across blocking reads and treats stop-induced close as success.Docs note the new behavior;
CloseReasonMonitorlabels dedicated monitor teardown. Regression tests use a fake TCP server for connection drop and stop-without-traffic.Reviewed by Cursor Bugbot for commit 4176bec. Bugbot is set up for automated code reviews on this repo. Configure here.