Backport of fix: prevent graceful shutdown errors from killing Envoy proxy immediately into release/2.0.x#1075
Merged
hc-github-team-consul-core merged 1 commit intorelease/2.0.xfrom Apr 29, 2026
Conversation
f6fb598 to
4e85cc7
Compare
github-team-consul-core-pr-approver
approved these changes
Apr 28, 2026
Collaborator
github-team-consul-core-pr-approver
left a comment
There was a problem hiding this comment.
Auto approved Consul Bot automated PR
santoshpulluri
approved these changes
Apr 29, 2026
panman90
added a commit
that referenced
this pull request
Apr 29, 2026
* Backport of fix: prevent graceful shutdown errors from killing Envoy proxy immediately into release/2.0.x (#1075) backport of commit 03475d3 Co-authored-by: santoshpulluri <santosh.pulluri@hashicorp.com> * Backport of fix: use net.JoinHostPort for remaining Envoy admin URLs into release/2.0.x (#1071) backport of commit fa7de22 Co-authored-by: santoshpulluri <santosh.pulluri@hashicorp.com> --------- Co-authored-by: hc-github-team-consul-core <github-team-consul-core@hashicorp.com> Co-authored-by: santoshpulluri <santosh.pulluri@hashicorp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This PR is auto-generated from #1072 to be assessed for backporting due to the inclusion of the label backport/2.0.
The below text is copied from the body of the original PR.
Problem
During graceful shutdown, any error from
DumpConfig(),Drain(), orQuit()callsclose(errorExitCh). This channel is monitored by theRun()select loop in consul_dataplane.go:This means a transient error during shutdown (e.g., Envoy not yet ready, network timeout, IPv6 URL issue) bypasses the entire grace period and kills Envoy at 0ms — exactly the symptom seen in
TestConnectInject_ProxyLifecycleShutdownacceptance test failures wherecurlreportsConnection refusedimmediately.Sequence of events (before fix)
Root cause
errorExitChis intended to signal that the lifecycle server itself has crashed (e.g.,ListenAndServefails). It should NOT be closed for transient errors during an already-in-progress graceful shutdown. The shutdown path should log errors and continue — the grace period must be honored regardless.Fix
Removed
close(errorExitCh)from all three error paths ingracefulShutdown():DumpConfig()errorDrain()errorQuit()errorErrors are still logged as warnings. The shutdown continues through the full grace period as intended.
Testing
Added 3 unit tests in lifecycle_test.go that reproduce the bug deterministically:
TestGracefulShutdown_DumpConfigError_DoesNotKillProxyDumpConfig()failure does not closeerrorExitChTestGracefulShutdown_DrainError_DoesNotKillProxyDrain()failure does not closeerrorExitChTestGracefulShutdown_QuitError_DoesNotKillProxyQuit()failure does not closeerrorExitChAlso fixed pre-existing issues in the test mock:
mockProxymethods now return error fields —Drain(),Quit(),DumpConfig()previously always returnednil, meaning error paths were never exercisedinttosync/atomic.Int32to eliminate races flagged bygo test -raceRelated PRs
skip_exittoDrain()+JoinHostPort(prevents Envoy self-terminating on drain)JoinHostPortforDumpConfig()+Ready()(prevents malformed IPv6 URLs)All three are needed together to fully fix
TestConnectInject_ProxyLifecycleShutdown.Overview of commits