Skip to content

Cancel connection backoff on graceful shutdown - #187

Merged
glbrntt merged 1 commit into
grpc:mainfrom
hamzahrmalik:cancel_backoff_on_shutdown
Jul 27, 2026
Merged

Cancel connection backoff on graceful shutdown#187
glbrntt merged 1 commit into
grpc:mainfrom
hamzahrmalik:cancel_backoff_on_shutdown

Conversation

@hamzahrmalik

Copy link
Copy Markdown
Contributor

Problem

When a gRPC client's connection is stuck in backoff (e.g. the server is unreachable), graceful shutdown would wait out that in-flight backoff sleep before returning — so shutdown could hang for up to the max backoff (120s by default), even with zero in-flight RPCs. In practice that means processes that are slow to exit on shutdown

Cause

A subchannel that is sleeping in connection backoff has no established connection and no in-flight RPCs to drain. Its backoff Task.sleep was spawned into the subchannel's discarding task group, which graceful shutdown awaits rather than cancels, so run() (and thereforeGRPCClient.runConnections()) did not return until the backoff elapsed — up to the default max of 120s — even with zero in-flight RPCs. This also contradicts runConnections()'s documented contract of returning once graceful shutdown is requested and in-flight RPCs have finished.

Changes

Spawn the backoff via addCancellableTask and store its handle on the .connecting state (mirroring how the name-resolver task handle is tracked). When shutDown() is called while backing off, cancel that task and finish immediately; when a connect attempt is genuinely in-flight, behaviour is unchanged.

The new test fails without this fix, and passes after it.

## Problem

When a gRPC client's connection is stuck in backoff (e.g. the server is unreachable), graceful shutdown would wait out that in-flight backoff sleep before returning — so shutdown could hang for up to the max backoff (120s by default), even with zero in-flight RPCs. In practice that means processes that are slow to exit on shutdown

## Cause

A subchannel that is sleeping in connection backoff has no established connection and no in-flight RPCs to drain. Its backoff `Task.sleep` was spawned into the subchannel's discarding task group, which graceful shutdown awaits rather than cancels, so `run()` (and therefore`GRPCClient.runConnections()`) did not return until the backoff elapsed — up to the default max of 120s — even with zero in-flight RPCs. This also contradicts `runConnections()`'s documented contract of returning once graceful shutdown is requested and in-flight RPCs have finished.

## Changes

Spawn the backoff via `addCancellableTask` and store its handle on the `.connecting` state (mirroring how the name-resolver task handle is tracked). When `shutDown()` is called while backing off, cancel that task and finish immediately; when a connect attempt is genuinely in-flight, behaviour is unchanged.

The new test fails without this fix, and passes after it.
@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 24, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: hamzahrmalik / name: Hamzah Malik (53cdcad)

@glbrntt glbrntt added the 🔨 semver/patch No public API change. label Jul 27, 2026

@glbrntt glbrntt left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @hamzahrmalik, nice patch.

@glbrntt
glbrntt merged commit 0cfa90d into grpc:main Jul 27, 2026
40 of 41 checks passed
dongjoon-hyun added a commit to apache/spark-connect-swift that referenced this pull request Aug 4, 2026
### What changes were proposed in this pull request?

This PR upgrades the `grpc-swift-nio-transport` dependency to `2.9.1`.

### Why are the changes needed?

To adopt the latest `grpc-swift-nio-transport` release (`2.9.1`, 2026-08-03).
- https://github.com/grpc/grpc-swift-nio-transport/releases/tag/2.9.1
  - grpc/grpc-swift-nio-transport#185
  - grpc/grpc-swift-nio-transport#187

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Fable 5

Closes #476 from dongjoon-hyun/SPARK-58542.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🔨 semver/patch No public API change.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants