fix(connection): retry entire connect+handshake to fix PubSub reconnect#304
Open
makubo-aws wants to merge 1 commit intovalkey-io:mainfrom
Open
fix(connection): retry entire connect+handshake to fix PubSub reconnect#304makubo-aws wants to merge 1 commit intovalkey-io:mainfrom
makubo-aws wants to merge 1 commit intovalkey-io:mainfrom
Conversation
Previously, only the socket-level _connect() was wrapped in the retry block. The on_connect() handshake (auth, protocol negotiation, etc.) ran outside the retry scope, so a ConnectionError raised during reconnection (e.g. 'Connection reset by peer' in PubSub) would propagate directly to the caller instead of being retried. This is the root cause of the PubSub retry bug reported in issue valkey-io#169 and the idle-to-burst ConnectionError bursts on ElastiCache Serverless reported in issue valkey-io#225. Fix: extract _connect_with_handshake() that combines the socket connect and on_connect() into a single retryable unit. The retry block now covers the full connection establishment flow. On failure, disconnect() is called as the error handler to ensure a clean state before the next attempt. The same fix is applied to both the sync (valkey/connection.py) and async (valkey/asyncio/connection.py) AbstractConnection classes. This mirrors the fix applied to redis-py in PR #3863. Fixes valkey-io#169 Related: valkey-io#225
makubo-aws
requested review from
ahmedsobeh,
bogdanp05,
mkmkme and
smeso
as code owners
Author
|
vibe coded based on existing fix for this in redis-py |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #304 +/- ##
===========================================
- Coverage 76.72% 62.23% -14.50%
===========================================
Files 129 129
Lines 34146 34150 +4
===========================================
- Hits 26199 21253 -4946
- Misses 7947 12897 +4950 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands connection retry coverage to include the full connect + handshake sequence (auth/protocol negotiation/client setup), addressing PubSub reconnect failures where handshake-time ConnectionErrors previously escaped the retry policy.
Changes:
- Wrap socket connect and
on_connect()handshake into a single retryable unit via a new_connect_with_handshake()helper (sync + async). - Update
connect()to retry_connect_with_handshake()and run connect callbacks after a successful connection. - Ensure cleanup between attempts by calling
disconnect()on failures.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
valkey/connection.py |
Wrap sync connect + handshake inside retry; add _connect_with_handshake() and move callbacks post-connect. |
valkey/asyncio/connection.py |
Mirror the same retry + handshake restructuring for asyncio connections. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
314
to
317
| await self.retry.call_with_retry( | ||
| lambda: self._connect(), lambda error: self.disconnect() | ||
| lambda: self._connect_with_handshake(), | ||
| lambda error: self.disconnect(), | ||
| ) |
Comment on lines
+321
to
324
| self.retry.call_with_retry( | ||
| lambda: self._connect_with_handshake(), | ||
| lambda error: self.disconnect(error), | ||
| ) |
Comment on lines
+360
to
362
| # clean up after any error in on_connect so that the next | ||
| # retry attempt starts from a clean state | ||
| self.disconnect() |
Comment on lines
+321
to
324
| self.retry.call_with_retry( | ||
| lambda: self._connect_with_handshake(), | ||
| lambda error: self.disconnect(error), | ||
| ) |
Comment on lines
314
to
317
| await self.retry.call_with_retry( | ||
| lambda: self._connect(), lambda error: self.disconnect() | ||
| lambda: self._connect_with_handshake(), | ||
| lambda error: self.disconnect(), | ||
| ) |
Comment on lines
360
to
364
| except ValkeyError: | ||
| # clean up after any error in on_connect | ||
| # clean up after any error in on_connect so that the next | ||
| # retry attempt starts from a clean state | ||
| await self.disconnect() | ||
| raise |
Collaborator
|
Could you have a look at:
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #169
Related: #225
Problem
The retry logic in
AbstractConnection.connect()only wrapped the socket-level_connect()call. Theon_connect()handshake (authentication, protocol negotiation, CLIENT SETNAME, etc.) ran outside the retry scope.When a PubSub connection drops and the client tries to reconnect,
on_connect()can raise aConnectionError(e.g.Connection reset by peer— errno 104) if the server isn't fully ready yet. Because this error escapes the retry block, the configured retry policy is never applied and the PubSub thread crashes instead of recovering.This is also the root cause of the idle-to-burst
ConnectionErrorbursts seen on ElastiCache Serverless (issue #225).Fix
Extract a
_connect_with_handshake()helper that combines the socket connect andon_connect()handshake into a single retryable unit. The retry block inconnect()now covers the full connection establishment flow. On failure,disconnect()is called as the error handler to ensure a clean state before the next attempt.The same fix is applied to both:
valkey/connection.py(sync)valkey/asyncio/connection.py(async)Prior art
This mirrors the fix applied to redis-py in PR #3863, which resolved the equivalent issues in that library.
Changes
valkey/connection.py:connect()now calls_connect_with_handshake()inside the retry block; new_connect_with_handshake()method combines socket connect + handshakevalkey/asyncio/connection.py: same changes for the async path