Skip to content

Wrap EOF/connection reset errors with TLS context after handshake#2031

Open
wallyqs wants to merge 1 commit intomainfrom
wq/lb-tls-eof
Open

Wrap EOF/connection reset errors with TLS context after handshake#2031
wallyqs wants to merge 1 commit intomainfrom
wq/lb-tls-eof

Conversation

@wallyqs
Copy link
Member

@wallyqs wallyqs commented Feb 27, 2026

When a TLS proxy (e.g. nginx with mTLS) completes the TLS handshake but then closes the connection due to an untrusted client certificate, the client would receive a bare "EOF" most of the time, making it difficult to to troubleshoot (since not sending tls alerts):

  ┌───────┬───────────────────────────────────────────────────────────────┬──────────────────────────────────────────┐
  │       │                            TLS 1.3                            │                 TLS 1.2                  │
  ├───────┼───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
  │ nginx │ Handshake succeeds → Read() returns EOF                       │ Handshake succeeds → Read() returns EOF  │
  ├───────┼───────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
  │ NATS  │ Handshake succeeds → Read() returns tls: certificate required │ Handshake fails → tls: handshake failure │
  └───────┴───────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘

Now we wrap these io.EOF errors with more context that this happened after the TLS handshake:

  "nats: connection closed by remote after TLS handshake: EOF"

The wrapped error preserves the original via %w for backwards compatibility (errors.Is(err, io.EOF) still returns true).

When a TLS proxy (e.g. nginx with mTLS) completes the TLS handshake but
then closes the connection due to an untrusted client certificate, the
client would receive a bare "EOF" or "connection reset by peer" with no
indication that TLS was involved. This wraps such errors with context:

  "nats: connection closed by remote after TLS handshake: EOF"

The wrapped error preserves the original via %w for backwards
compatibility (errors.Is(err, io.EOF) still returns true).

Signed-off-by: Waldemar Quevedo <wally@nats.io>
Copy link
Member

@philpennock philpennock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from a crypto/security perspective.

}
// Should contain a TLS-related error message.
errStr := err.Error()
if !strings.Contains(errStr, "tls:") && !strings.Contains(errStr, "certificate") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there any TLS errors which contain certificate but not tls:?
I know that x509: is another prefix in certain circumstances, depending upon the nature of the failure, but don't know which errors might trigger the second check to match but not the first.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, think we can just check tls: here

return err
}
if errors.Is(err, io.EOF) || isConnClosedError(err) {
return fmt.Errorf("nats: connection closed by remote after TLS handshake: %w", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only saying this in case you haven't considered it: perhaps nats: tls: ... as a prefix so that tls: checks will match?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haven't considered it, is that to help doing sttring.Contains on the tls: prefix? wdyt should we add it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@piotrpio wdyt about a good error string here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants