clientconn: Wait for all goroutines on close #8666

twz123 · 2025-10-21T20:34:32Z

Three goroutines could outlive a call to ClientConn.close(). Add mechanics to cancel them and wait for them to complete when closing a client connection.

Fixes #8655.

RELEASE NOTES:

Closing a client connection will cancel all pending goroutines and block until they complete.

codecov · 2025-10-21T20:38:20Z

Codecov Report

❌ Patch coverage is 89.23077% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.22%. Comparing base (25dbb81) to head (11b448a).
⚠️ Report is 8 commits behind head on master.

Files with missing lines	Patch %	Lines
clientconn.go	80.00%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8666      +/-   ##
==========================================
- Coverage   83.29%   83.22%   -0.07%     
==========================================
  Files         414      414              
  Lines       32753    32793      +40     
==========================================
+ Hits        27280    27291      +11     
- Misses       4070     4092      +22     
- Partials     1403     1410       +7

Files with missing lines	Coverage Δ
balancer_wrapper.go	`84.91% <100.00%> (+1.50%)`	⬆️
internal/testutils/pipe_listener.go	`84.61% <100.00%> (+0.83%)`	⬆️
test/rawConnWrapper.go	`66.10% <100.00%> (+0.28%)`	⬆️
clientconn.go	`90.27% <80.00%> (+0.30%)`	⬆️

... and 21 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

eshitachandwani · 2025-11-04T09:04:58Z

balancer_wrapper.go

 	healthData *healthData
+
+	shutdownMu    sync.Mutex
+	shutdown      chan struct{}


Could you change this to shutdownCh to indicate it is a channel?

eshitachandwani · 2025-11-04T19:52:42Z

balancer_wrapper.go


 func (acbw *acBalancerWrapper) UpdateAddresses(addrs []resolver.Address) {
-	acbw.ac.updateAddrs(addrs)
+	acbw.goFunc(func(shutdown <-chan struct{}) {


Is there a reason we changed this to run the acbw.ac.updateAddrs(addrs) function in a go routine??

It's basically a "bubbled-up" goroutine. Previously, the goroutine was spawned in updateAddrs itself (line 1021). But, as we now need to track those, I figured it would be most appropriate to do it here. Another option would be to somehow push this down into updateAddrs itself, by passing the acBalancerWrapper pointer, or a function pointer to acbw.goFunc or sth. along those lines, and then use that to spawn the goroutine there:

Suggested change

acbw.goFunc(func(shutdown <-chan struct{}) {

acbw.ac.updateAddrs(acbw, addrs)

Then we could write line 1021 of updateAddrs like so:

acbw.goFunc(ac.resetTransportAndUnlock)

I've changed it this way. Seems to be clearer.

eshitachandwani · 2025-11-05T09:46:04Z

clientconn.go

-	ac.mu.Lock()
-	defer ac.mu.Unlock()
+	acMu := &ac.mu
+	acMu.Lock()


Is there a reason the original code did not work? this seems unnecessarily complicated and does the same thing. Or am I missing something.

It's a way to make the defer do the unlock conditionally, as the code might need to unlock it before returning (see lines 1441 and 1442). We can achieve the same e.g. with a boolean variable, if you prefer.

I think using a pointer for this is good, because, if you forget the nil check, it will panic with an easy to understand stack trace. Whereas if you forget to check a boolean, you'd do a double-unlock and there's a chance that you get weirder problems.

internal/balancer/gracefulswitch/gracefulswitch.go

easwars · 2025-12-02T21:52:20Z

@twz123
We had a discussion about this PR during our GitHub issues/PR scrub. Can I request you to split these into separate PRs. The change to the graceful_switch LB policy is much simpler compared to the changes in the clientconn for the other two goroutine leaks. We want to make progress on this as much as possible, but are also a little concerned about the complexity that it adds to the clientconn code (which is already reasonably complex).

Apologies for how long this review has dragged on, but hopefully once these are split into smaller ones, we should be able to move faster.

Thanks for your understanding.

twz123 · 2025-12-05T11:15:33Z

@twz123 We had a discussion about this PR during our GitHub issues/PR scrub. Can I request you to split these into separate PRs. The change to the graceful_switch LB policy is much simpler compared to the changes in the clientconn for the other two goroutine leaks. We want to make progress on this as much as possible, but are also a little concerned about the complexity that it adds to the clientconn code (which is already reasonably complex).

Done: #8746

Apologies for how long this review has dragged on, but hopefully once these are split into smaller ones, we should be able to move faster.

Thanks for your understanding.

No worries. I understand that these things take time.

Goroutines spawned during balancer swaps could outlive the call to Balancer.Close(). Monitor these via a wait group and wait for them to finish before returning from Close(). This prevents any noticeable side effects that could otherwise occur after Close() returns. See: * #8655 * #8666 (comment) RELEASE NOTES: - client: Closing a graceful switch balancer will now block until all pending goroutines complete. Signed-off-by: Tom Wieczorek <[email protected]>

github-actions · 2025-12-28T16:49:22Z

This PR is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

twz123 · 2025-12-30T08:34:45Z

Not stale.

github-actions · 2026-01-05T08:52:58Z

This PR is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

The health check goroutine could outlive a call to ClientConn.close(). Add a done channel that will be waited on when closing the transport. RELEASE NOTES: - Closing a client connection will now block until the health check goroutine completes. Signed-off-by: Tom Wieczorek <[email protected]>

twz123 · 2026-01-05T09:09:08Z

This is not stale. There's other PRs pending before I can un-draft this one again.

This will properly pass on cancellation requests, and reduce usage of a deprecated method. Signed-off-by: Tom Wieczorek <[email protected]>

When a connection attempt is canceled after the transport is created, ensure that it is closed inline. This prevents untracked goroutines from being left behind. See: * #8655 * #8666 (comment) RELEASE NOTES: * When canceling a connection attempt, closing the transport won't use an untracked goroutine anymore. Signed-off-by: Tom Wieczorek <[email protected]>

See: * #8666 (comment) RELEASE NOTES: none Signed-off-by: Tom Wieczorek <[email protected]>

github-actions · 2026-01-11T12:59:54Z

This PR is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

…tdialer' into clientconn-close-waitfor-goroutines

Three goroutines could outlive a call to ClientConn.close(). Add mechanics to cancel them and wait for them to complete when closing a client connection. RELEASE NOTES: - Closing a client connection will cancel all pending goroutines and block until they complete. Signed-off-by: Tom Wieczorek <[email protected]>

github-actions · 2026-01-19T14:51:53Z

This PR is labeled as requiring an update from the reporter, and no update has been received after 6 days. If no update is provided in the next 7 days, this issue will be automatically closed.

eshitachandwani added this to the 1.77 Release milestone Oct 22, 2025

eshitachandwani added the Type: Bug label Oct 22, 2025

eshitachandwani self-requested a review October 22, 2025 08:26

eshitachandwani self-assigned this Oct 22, 2025

eshitachandwani requested a review from easwars October 27, 2025 07:45

eshitachandwani assigned easwars Oct 27, 2025

arjan-bal modified the milestones: 1.77 Release, 1.78 Release Oct 30, 2025

eshitachandwani reviewed Nov 5, 2025

View reviewed changes

eshitachandwani assigned twz123 and unassigned eshitachandwani Nov 5, 2025

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from e45e9d7 to e8bb2b4 Compare November 5, 2025 12:28

easwars reviewed Nov 13, 2025

View reviewed changes

internal/balancer/gracefulswitch/gracefulswitch.go Outdated Show resolved Hide resolved

internal/balancer/gracefulswitch/gracefulswitch.go Outdated Show resolved Hide resolved

internal/balancer/gracefulswitch/gracefulswitch.go Outdated Show resolved Hide resolved

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from e8bb2b4 to 3fa606c Compare November 20, 2025 08:10

eshitachandwani unassigned twz123 Nov 25, 2025

easwars assigned twz123 and unassigned easwars Dec 2, 2025

easwars added the Status: Requires Reporter Clarification label Dec 2, 2025

twz123 mentioned this pull request Dec 5, 2025

gracefulswitch: Wait for all goroutines on close #8746

Merged

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from 3fa606c to 00deb21 Compare December 5, 2025 11:12

twz123 marked this pull request as draft December 5, 2025 11:15

eshitachandwani modified the milestones: 1.78 Release, 1.79 Release Dec 11, 2025

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from 00deb21 to 40605a5 Compare December 12, 2025 14:37

twz123 marked this pull request as ready for review December 12, 2025 14:47

eshitachandwani unassigned twz123 Dec 16, 2025

twz123 force-pushed the clientconn-close-waitfor-goroutines branch 2 times, most recently from ad9c99a to e8b6f4d Compare December 22, 2025 14:44

github-actions bot added the stale label Dec 28, 2025

github-actions bot removed the stale label Dec 30, 2025

github-actions bot added the stale label Jan 5, 2026

arjan-bal added stale and removed stale labels Jan 5, 2026

twz123 mentioned this pull request Jan 5, 2026

test/channelz: Don't try to dial non-dialable addresses #8782

Draft

test: Replace WithDialer usages with WithContextDialer

7965ceb

This will properly pass on cancellation requests, and reduce usage of a deprecated method. Signed-off-by: Tom Wieczorek <[email protected]>

github-actions bot removed the stale label Jan 5, 2026

eshitachandwani pushed a commit that referenced this pull request Jan 8, 2026

transport: Remove unused return value from connect() (#8787)

85ede8e

See: * #8666 (comment) RELEASE NOTES: none Signed-off-by: Tom Wieczorek <[email protected]>

github-actions bot added the stale label Jan 11, 2026

Merge branches 'clientconn-wait-for-healthcheck' and 'test-withcontex…

b73adc3

…tdialer' into clientconn-close-waitfor-goroutines

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from e8b6f4d to f69b1fd Compare January 13, 2026 13:14

twz123 force-pushed the clientconn-close-waitfor-goroutines branch from f69b1fd to 11b448a Compare January 13, 2026 13:38

github-actions bot removed the stale label Jan 13, 2026

github-actions bot added stale and removed stale labels Jan 19, 2026

eshitachandwani removed the Status: Requires Reporter Clarification label Jan 19, 2026

Pranjali-2501 modified the milestones: 1.79 Release, 1.80 Release Jan 21, 2026

	acbw.goFunc(func(shutdown <-chan struct{}) {
	acbw.ac.updateAddrs(acbw, addrs)

clientconn: Wait for all goroutines on close #8666

Are you sure you want to change the base?

clientconn: Wait for all goroutines on close #8666

Uh oh!

Conversation

twz123 commented Oct 21, 2025

Uh oh!

codecov bot commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

eshitachandwani Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

twz123 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

eshitachandwani Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

twz123 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

twz123 Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

eshitachandwani Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

twz123 Nov 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

easwars commented Dec 2, 2025

Uh oh!

twz123 commented Dec 5, 2025

Uh oh!

github-actions bot commented Dec 28, 2025

Uh oh!

twz123 commented Dec 30, 2025

Uh oh!

github-actions bot commented Jan 5, 2026

Uh oh!

twz123 commented Jan 5, 2026

Uh oh!

github-actions bot commented Jan 11, 2026

Uh oh!

github-actions bot commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Oct 21, 2025 •

edited

Loading