Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via etcd maintainers.
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

We use grpc-gateway and grpc-proxy to reduce watch requests from apisix to etcd, as apisix only provides http client for etcd for production use. Here is a simple architecture diagram:

The grpc-gateway is a simple program compiled by the code below:

package main

import (
	"context"
	"fmt"
	"golang.org/x/net/http2"
	"google.golang.org/grpc/credentials/insecure"
	"math"
	"net/http"
	"os"

	"github.com/grpc-ecosystem/grpc-gateway/runtime"
	"github.com/tmc/grpc-websocket-proxy/wsproxy"
	etcdservergw "go.etcd.io/etcd/api/v3/etcdserverpb/gw"
	"go.uber.org/zap"
	"google.golang.org/grpc"
	"google.golang.org/grpc/grpclog"
)

type wsProxyZapLogger struct {
	*zap.Logger
}

func (w wsProxyZapLogger) Warnln(i ...any) {
	w.Warn(fmt.Sprint(i...))
}

func (w wsProxyZapLogger) Debugln(i ...any) {
	w.Debug(fmt.Sprint(i...))
}

func main() {
	ctx := context.Background()
	ep := os.Getenv("ETCD")
	grpclog.SetLoggerV2(grpclog.NewLoggerV2(os.Stdout, os.Stderr, os.Stderr))

	opts := []grpc.DialOption{grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(math.MaxInt32))}
	opts = append(opts, grpc.WithTransportCredentials(insecure.NewCredentials()))

	conn, err := grpc.DialContext(ctx, ep, opts...)
	if err != nil {
		panic(err)
	}
	defer conn.Close()
	gwmux := runtime.NewServeMux()

	type registerHandlerFunc func(context.Context, *runtime.ServeMux, *grpc.ClientConn) error

	handlers := []registerHandlerFunc{
		etcdservergw.RegisterKVHandler,
		etcdservergw.RegisterWatchHandler,
		etcdservergw.RegisterLeaseHandler,
		etcdservergw.RegisterClusterHandler,
		etcdservergw.RegisterMaintenanceHandler,
		etcdservergw.RegisterAuthHandler,
	}
	for i := range handlers {
		if err = handlers[i](ctx, gwmux, conn); err != nil {
			panic(err)
		}
	}
	httpmux := http.NewServeMux()
	httpmux.Handle(
		"/v3/",
		wsproxy.WebsocketProxy(
			gwmux,
			wsproxy.WithRequestMutator(
				// Default to the POST method for streams
				func(_ *http.Request, outgoing *http.Request) *http.Request {
					outgoing.Method = "POST"
					return outgoing
				},
			),
			wsproxy.WithMaxRespBodyBufferSize(0x7fffffff),
			wsproxy.WithLogger(wsProxyZapLogger{}),
		),
	)
	srv := http.Server{
		Handler: httpmux,
		Addr:    "0.0.0.0:8887",
	}
	http2.ConfigureServer(&srv, &http2.Server{
		MaxConcurrentStreams: 16,
	})
	srv.ListenAndServe()
}

Start etcd(v3.5.13) and grpc-proxy locally by commands below:

nohup etcd&
nohup etcd grpc-proxy start --endpoints=localhost:2379 &

Then start grpc-gateway locally(compiled by the code above).
Send watch command to grpc-gateway with curl:

curl -X POST -d '{"create_request": {"key":"dGVzdGtleQo=","range_end":"dGVzdGtleTEK"}}' 'localhost:8887/v3/watch'

The curl command returns directly instead of waiting for watch response,with the error repsose below:

{"result":{"header":{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"created":true}} {"error":{"grpc_code":1,"http_code":408,"message":"context canceled","http_status":"Request Timeout"}}

We set auth to etcd by command below:

etcdctl user add root
etcdctl auth enable

Then make a kv range call with curl:

curl -X POST -d '{"key": "dGVzdGtleQo=", "range_end": "dGVzdGtleTEK"}' http://localhost:8887/v3/kv/range

The response is as expected:

{"error":"etcdserver: user name is empty","code":2,"message":"etcdserver: user name is empty"}

However, after we get a token with the command below:

curl 'localhost:8887/v3/auth/authenticate' -X POST -d '{"name":"root","password":"123456"}
{"header"{"cluster_id":"14841639068965178418","member_id":"10276657743932975437","revision":"1","raft_term":"2"},"token":"ImhAoKYyjbUPSKoN.11"}

We make the same kv range call with auth header:

curl -H 'Authorization: ImhAoKYyjbUPSKoN.11' -X POST -d '{"key": "dGVzdGtleQo=", "range_end": "dGVzdGtleTEK"}' http://localhost:8887/v3/kv/range

We still get the user name is empty error.

{"error":"etcdserver: user name is empty","code":2,"message":"etcdserver: user name is empty"}

The expected response is invalid auth token error:

{"error":"etcdserver: invalid auth token","code":2,"message":"etcdserver: invalid auth token"}

After some digging in the source code, we found the grpc server of grpc proxy in etcd did not propagate grpc metadata properly.
And the watch cancled error may caused by the client early disconnection by grpc-gateway.
We propose a PR trying to fix the problem, expecting for reviewing.

What did you expect to happen?

Grpc proxy can handle grpc gateway requests correctly.

How can we reproduce it (as minimally and precisely as possible)?

As describe above.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.13
Git SHA: 6bbccf4da
Go Version: go1.22.2
Go OS/Arch: darwin/amd64

$ etcdctl version
etcdctl version: 3.5.13
API version: 3.5

Etcd configuration (command line flags or environment variables)

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response