Transport-level retry support for RPCs with streaming request

[jhump]

With HTTP/2, when a server is performing graceful shutdown, it sends a GOAWAY frame to a client, if the client tries to start a new stream. During this shutdown, the server refuses new streams and will continue processing existing streams until they are complete (or until some timeout occurs).

With a connect-go client, this can result in errors like the following:

unavailable: http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

PR #649 addresses the above issues when it comes to handling unary requests (unary and server-stream RPCs). But it's still a possible issue with streaming requests.

Implementing GetBody with a streaming request is more complicated because it will involve buffering some (likely small) amount of request data, as the client is uploading, which can then be "replayed" in the event of a transport-level retry and use of GetBody. This issue is to track the state of such a future enhancement.

[aimuz]

http.Server.Shutdown does not achieve graceful exit for GPRC, especially in cases where certain GRPC clients maintain long connections. This issue is particularly severe, and the server encounters this problem every time it rolls out service updates.

In the implementation of grpc, a goAway is sent to notify the client, but this mechanism has not been found in connectrpc.

[jhump]

In the implementation of grpc, a goAway is sent to notify the client, but this mechanism has not been found in connectrpc.

That's because connectrpc does not implement HTTP/2: that all happens inside of the standard "net/http" package. If you go through graceful shutdown of the http.Server (which is admittedly non-trivial), it should go through the same process and send a GOAWAY frame to the client.

[aimuz]

Yes, a normal http2 server can do this, but it seems that it cannot be directly supported under h2c.

[aimuz]

	h2 := &http2.Server{}
	server := &http.Server{
		Addr:    addr,
		Handler: h2c.NewHandler(withCORS(s.mux), h2),
	}

	err := http2.ConfigureServer(server, h2)
	if err != nil {
		return err
	}

A simple test was done. This method will use GOAWAY frames. Maybe it should be updated in the following document:

https://connectrpc.com/docs/go/deployment#h2c

[aimuz]

golang/go#26682

There is some new content that doesn't seem to shut down gracefully.

[jhump]

@aimuz, in that example snippet, are you saying that http2.ConfigureServer with a separate *http2.Server is required to get shutdown behave in the desired way?

As for the bug, I assume you saw the recent reply: Go 1.24 now includes a way to use "h2c" without having to import golang.org/x/net/http2, and it should behave in the desired way. We definitely have a documentation task to update the sections of Connect docs that talk about h2c to refer to the new mechanism if using Go 1.24 and above.