Skip to content

[PureGo] Preserve server rejection on handshake timeout - #636

Open
zlata-stefanovic-db wants to merge 6 commits into
mainfrom
fix/purego-handshake-status-635
Open

[PureGo] Preserve server rejection on handshake timeout#636
zlata-stefanovic-db wants to merge 6 commits into
mainfrom
fix/purego-handshake-status-635

Conversation

@zlata-stefanovic-db

@zlata-stefanovic-db zlata-stefanovic-db commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

  • preserve a terminal server status in the raw_stream handshake expiry path instead of always returning hctx.Err()
  • decide the reported cause on hctx's state rather than on which select case won, so a response and an expiry becoming ready together cannot make the error a coin flip
  • call teardown whenever hctx has expired, including when the recv branch observed the result first
  • exclude cancellation-shaped outcomes (isTeardownArtifact) so an RPC the SDK itself cancelled is not misread as a server rejection
  • add regression coverage for both directions of the rule, plus a determinism check

Why

When the server's rejection lands in the same window as the handshake deadline,
the status was discarded and Open returned a bare DeadlineExceeded. Two
things follow from losing the code:

  • isAuthRejection(err) is false, so HeadersProvider.Invalidate is never
    called and rejected credentials stay cached.
  • The stream layer sees a context error. isRetryable in
    internal/stream/supervisor.go treats context.DeadlineExceeded as
    terminal, so the supervisor breaks its recovery loop, fails the watermark,
    and abandons every buffered record to OnError. A permanent setup failure that
    a fresh token would fix instead kills the stream outright.

Note for anyone reading #635: its impact section predicts the opposite outcome —
that a transport.IsTerminalStatus returns false and the supervisor "retries for
the full recovery budget, resending records each time". That function does not
exist, and the stream layer does no gRPC-code classification at all. The real
behaviour is terminal-on-first-occurrence, which is worse than the issue
describes.

Why the exclusion list is necessary rather than a heuristic

An earlier revision of this PR tried to replace the shape-based exclusions with
ordering: trust any result reaped before teardown(), on the theory that it
must predate the cancellation and therefore came from the server.
TestOpenHonorsCallerDeadline rejects that. Conn.Open installs a second
canceller — context.AfterFunc(handshakeCtx, cancelStream) — which fires as soon
as handshakeCtx expires, independently of handshake's own teardown. A
codes.Canceled can already be sitting in done before teardown is called, so
arrival order says nothing about provenance and the error's shape is the only
signal available.

Scope

This makes the reported cause faithful; it does not change the fact that an
expired hctx always fails the handshake, and it does not resurrect a readiness
response that arrived too late (Open already fails that case via its
post-handshake bridge check).

It also does not address the related defect in #637: because isRetryable treats
any context.DeadlineExceeded as terminal, an internal setup budget expiring is
itself misclassified as permanent. Fixing that would make this class of masking
self-healing — the next attempt would surface the real status and re-mint — which
is how the Rust core already behaves (DeadlineExceeded is absent from
UNRETRIABLE_STATUS_CODES). The two changes are complementary and independent.

Fixes #635

Testing

  • gofmt -l ., go vet ./...
  • go test -race -count=3 ./... from purego/
  • new: TestRawStreamHandshakeTimeoutPrefersServerRejection (rejection replaces
    DeadlineExceeded), TestRawStreamHandshakeTimeoutKeepsDeadlineForTeardownArtifacts
    (table over gRPC Canceled/DeadlineExceeded, bare context errors, and EOF),
    TestRawStreamHandshakeTimeoutRejectionIsDeterministic (200 iterations across
    both select interleavings)

Prefer a real server rejection over deadline/cancel in rawStream handshake timeout path.

Adds regression test for teardown-gated Recv returning Unauthenticated.

Fixes #635

Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
@zlata-stefanovic-db zlata-stefanovic-db self-assigned this Jul 29, 2026
@zlata-stefanovic-db
zlata-stefanovic-db requested a review from a team July 29, 2026 14:24
Move the rejection-vs-deadline decision after the select so it no longer
depends on which case wins the race, guarantee teardown whenever hctx has
expired, and extract the classification as isTeardownArtifact.

Add coverage for the teardown-artifact branches (gRPC and bare context
cancellation, EOF), assert the rejection replaces DeadlineExceeded rather
than merely being present, and consolidate the two near-identical blocking
RPC fakes.

Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
@zlata-stefanovic-db
zlata-stefanovic-db marked this pull request as ready for review July 29, 2026 15:35

@teodordelibasic-db teodordelibasic-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing, unrelated to PR, at purego/internal/transport/transport_test.go:529:

The regression suite does not exercise the user-visible failure path where an auth rejection has already been captured, the open context expires, and HeadersProvider.Invalidate must still run. The new handshake tests cover arbitration with fakes, while this existing test covers only an immediate rejection without deadline pressure, so the AfterFunc bridge, gRPC wrapping, and invalidation are never exercised together.

Could we add a deterministic Conn.Open test with a client-stream interceptor that first captures the real RecvMsg result and then holds it until openCtx.Done() before returning it?

func (s *heldRecvErrorStream) RecvMsg(m any) error {
        err := s.ClientStream.RecvMsg(m)
        <-s.release // openCtx.Done()
        return err
}

With the in-memory server returning codes.Unauthenticated, this guarantees the trailer is captured before the deadline but reaches the handshake only after hctx has expired. The test can assert that Open returns codes.Unauthenticated, not context.DeadlineExceeded, and that Invalidate is called once.

Comment thread purego/internal/transport/raw_stream.go Outdated
// error instead, and EOF carries no status at all.
func isTeardownArtifact(err error) bool {
switch status.Code(err) {
case codes.Canceled, codes.DeadlineExceeded:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

codes.DeadlineExceeded is not a teardown artifact on the current Conn.Open path. streamCtx is built with context.WithoutCancel and context.WithCancel, so it has no deadline. Calling cancelStream makes grpc-go report codes.Canceled; a codes.DeadlineExceeded returned by Recv can therefore be a server status.

Treating that status as teardown noise replaces it with hctx.Err(). This also changes recovery behavior today: isRetryable treats the context error as terminal, while a gRPC status with the same code does not satisfy errors.Is(err, context.DeadlineExceeded) and remains retryable. Could we narrow the gRPC case to codes.Canceled and keep the bare context checks below it?

switch status.Code(err) {
case codes.Canceled:
        return true
}

The existing grpc deadline exceeded table case can then become a preservation case using serverErr := status.Error(codes.DeadlineExceeded, "server deadline") and asserting errors.Is(got, serverErr). That assertion deterministically fails with the current classifier and passes after the switch is narrowed.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented both, thanks for the detailed suggestions

Added a deterministic Conn.Open regression using a held RecvMsg interceptor, so we now cover the “auth rejection captured first, open deadline expires before return” race. The test asserts we return Unauthenticated (not DeadlineExceeded) and still call Invalidate once.

Also narrowed teardown-artifact gRPC classification to codes.Canceled only. codes.DeadlineExceeded is now preserved as a possible server status, and the tests were updated to assert that preservation path.

Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Signed-off-by: Zlata Stefanovic <zlata.stefanovic@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[PureGo] Handshake deadline discards the server's rejection status

2 participants