Skip to content

fix(connection): fail fast on unrecoverable connect errors (#4026) - #4166

Open
sean-kim05 wants to merge 1 commit into
redis:masterfrom
sean-kim05:fix/connect-fast-fail-unrecoverable
Open

fix(connection): fail fast on unrecoverable connect errors (#4026)#4166
sean-kim05 wants to merge 1 commit into
redis:masterfrom
sean-kim05:fix/connect-fast-fail-unrecoverable

Conversation

@sean-kim05

@sean-kim05 sean-kim05 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Addresses the narrow, separable problem I raised in #4026: definitively-unrecoverable connection errors are retried with full backoff. The intentional handshake-phase retry behavior (clarified by @petyaslavova in that thread) is left completely untouched.

Since 7.1.0, Connection.connect() wraps the entire connect + handshake in the retry policy. During socket establishment every OSError is re-wrapped into a generic ConnectionError; because the policy's supported errors include ConnectionError, these are retried up to DEFAULT_RETRY_COUNT (10) times with exponential backoff — even for errors that can never succeed on retry, e.g. a Unix-domain-socket ENOENT (the socket path simply doesn't exist). The wrap also discarded the original errno, so nothing downstream could distinguish a permanent failure from a transient one.

Net effect: Redis(unix_socket_path=..., socket_connect_timeout=2) against a missing socket spends seconds sleeping/retrying, and the common "try UDS, fall back to TCP" pattern becomes very slow.

Change

Uses the existing is_retryable hook in Retry.call_with_retry that connect() simply wasn't passing:

  1. Preserve the original error — the OSErrorConnectionError wrap now uses raise ... from e, so the original errno survives for classification.
  2. Pass an is_retryable predicate from connect() that returns False for a conservative set of unrecoverable socket errnos (currently just ENOENT). Those fail fast; transient errors such as ECONNREFUSED during server startup keep retrying exactly as before.

Applied to both the sync (redis/connection.py) and async (redis/asyncio/connection.py) stacks.

Tests

  • test_connect_without_retry_on_unrecoverable_oserror (sync + async): ENOENT → single attempt, original OSError preserved as __cause__.
  • test_connect_retries_on_transient_oserror (sync + async): ECONNREFUSED → still retried (3 attempts with retries=2).

Open questions (carried over from #4026)

Kept deliberately conservative — happy to adjust to your preference:

  1. Scope of "unrecoverable": just ENOENT for now — should it also include EACCES / EAFNOSUPPORT / EPROTONOSUPPORT?
  2. socket_connect_timeout as a total-time cap: left out of scope here (a broader change carrying a deadline across retries); could be a follow-up.
  3. Default vs opt-in: implemented as the default; happy to gate behind a flag if preferred.

Refs #4026.


Note

Low Risk
Narrow connect-path retry behavior change with conservative errno list and tests; handshake retry logic is unchanged.

Overview
Connect failures that cannot succeed on retry (e.g. missing Unix socket, ENOENT) now fail immediately instead of burning the default retry count and exponential backoff.

Sync and async Connection.connect() pass is_retryable=self._is_retryable_connect_error into the existing Retry.call_with_retry hook. The predicate treats errors as non-retryable when the raised ConnectionError chains an OSError whose errno is in _UNRETRYABLE_CONNECT_ERRNOS (currently ENOENT only). ECONNREFUSED and other transient errors keep retrying as before.

OSError handling during connect now uses raise ConnectionError(...) from e so the original errno is available on __cause__ for that classification. Tests cover one-shot failure on ENOENT and full retries on ECONNREFUSED.

Reviewed by Cursor Bugbot for commit 3459a64. Bugbot is set up for automated code reviews on this repo. Configure here.

connect() wrapped the whole connect+handshake in the retry policy and re-wrapped every OSError into a generic ConnectionError, so definitively unrecoverable failures such as a missing Unix socket path (ENOENT) were retried DEFAULT_RETRY_COUNT times with full exponential backoff.

Pass an is_retryable predicate from connect() that returns False for a conservative set of unrecoverable errnos (currently ENOENT), and preserve the original OSError as the exception cause (raise ConnectionError(...) from e) so the retry policy can classify it. Transient errors like ECONNREFUSED during server startup remain retryable, so the intentional handshake-retry behavior is unchanged.

Mirrored across the sync and async connection stacks, with tests in both.
@petyaslavova

Copy link
Copy Markdown
Collaborator

Hi @sean-kim05, thank you for your contribution! I'll review it shortly.

Comment thread redis/connection.py

# OS-level errno values for which reconnecting can never succeed, so
# there is no point spending the retry/backoff budget on them.
_UNRETRYABLE_CONNECT_ERRNOS = frozenset({errno.ENOENT})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ENOENT is transient for a Unix socket while the server is starting: the path does not exist until the server binds it. This makes an explicitly configured retry stop after one attempt in that common startup race; should this behavior be opt-in rather than overriding every caller’s Retry policy?

@petyaslavova petyaslavova left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the careful work on this, and for keeping the handshake-retry behavior intact.

Before we merge, I'd like to apply some changes on the design toward making this configurable on the Retry object rather than hardcoding a non-retryable errno set as default behavior in the connection layer.

Concretely:

  1. Promote the original error up to the retry logic. Keep the raise ConnectionError(...) from e change so the root cause (and its errno) is available, and please make cause preservation consistent across the retryable wrap sites in connection.py (the socket read/write wraps currently build ConnectionError from e.args without from e, so __cause__ is lost there). Any cause-inspecting checker needs the root error to actually be reachable.

  2. Add a new optional property on the Retry object — a custom "is this error retryable?" checker. When set, it is applied on top of the existing supported-errors check and composed as an AND: an error is retried only if it is one of the supported error types AND the custom checker returns True. When it is not set, behavior is unchanged.

  3. Make it opt-in. We don't want to change the default retry behavior; the ENOENT/unrecoverable-errno logic should be something a user configures via this checker (we can ship it as a reusable helper), not an always-on default. This also means the classification then applies uniformly to every retry site that uses the Retry object, including the lazy reconnect path, instead of only the explicit connect() call.

Please mirror this in both the sync and async Retry/connection stacks, and add regression tests that fail on base and pass on the branch — including a real (non-mock) missing-socket path. One scoping note: this addresses the ENOENT case, not the original #4026 report (an unavailable TCP server / socket_connect_timeout as a total cap), so let's keep #4026 open and retitle this PR accordingly. Once the checker lives on the Retry object and composes as described, this should be ready for another review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants