metadata.FromOutgoingContext is too slow

[maralla]

The pr #6765 made the metadata.FromOutgoingContextRaw to be internal only. And now the only way to get outgoing metdata is metadata.FromOutgoingContext. But here is the benchmark:

goos: linux
goarch: amd64
pkg: md
cpu: 11th Gen Intel(R) Core(TM) i7-11700 @ 2.50GHz
BenchmarkFromOutgoingContext-16          1000000              1172 ns/op             496 B/op          8 allocs/op
BenchmarkFromOutgoingContextRaw-16      288412128                3.989 ns/op           0 B/op          0 allocs/op
PASS
ok      md      2.753s

from the following code:

func prepareData() context.Context {
        ctx := metadata.NewOutgoingContext(context.TODO(), metadata.New(map[string]string{
                "foo-bar":                              "bar",
                "61e61f68-8a36-4ca8-b993-079210836049": "123",
                "bfe9968be134":                         "foo",
                "36ad":                                 "bar",
        }))

        ctx = metadata.AppendToOutgoingContext(ctx, "df63a9e9", "1")
        ctx = metadata.AppendToOutgoingContext(ctx, "350e070d4958", "2")

        return ctx
}

func BenchmarkFromOutgoingContext(b *testing.B) {
        ctx := prepareData()

        b.ReportAllocs()

        var md metadata.MD

        for i := 0; i < b.N; i++ {
                md, _ = metadata.FromOutgoingContext(ctx)
        }

        if md.Get("foo-bar")[0] != "bar" {
                b.Error(md)
        }
}

func BenchmarkFromOutgoingContextRaw(b *testing.B) {
        ctx := prepareData()

        b.ReportAllocs()

        var md metadata.MD

        for i := 0; i < b.N; i++ {
                md, _, _ = metadata.FromOutgoingContextRaw(ctx)
        }

        if md.Get("foo-bar")[0] != "bar" {
                b.Error(md)
        }
}

Compare to FromOutgoingContextRaw the function FromOutgoingContext is very slow which makes it no easy way to get metadata with no overhead. Our projects (internal only) rely heavily on metadata handling. I'm asking, do we have any chance to add back the "raw" function? Or, does it have any plan to make FromOutgoingContext fast?

[easwars]

Our projects (internal only) rely heavily on metadata handling

Could you shed some more light on the "how/why/what" of metadata handling you do in your projects?

It would also be good to see how much performance impact this is having in real life (as opposed to a micro-benchmark).

Thanks

[maralla]

Our company uses go-grpc for all our internal micro-services which are business heavy domains, like consumer facing community system, payment system, marketing system, etc. These services utilize metadata for passing many extra information, some are helpful auxiliary key-value data (tracing, routing, etc.), some are highly related to logic behaviors. And as our internal service mesh is relatively complex, each intermediate service may inspect some metadata, and/or add more info, then pass to the next hop, the metadata may accumulated to a considerable size. We actually tried changing to use the currently existing method FromOutgoingContext, immediately, one of our business system found a non-trivial performance regression through their routinely scheduled stress test, then we rolled back.

To some extent, we may have been misusing metadata, and should instead simply put many fields in request or response messages. However, at the very beginning when the "raw" method exists, we chose it as a lightweight way of passing dynamic or extra information through the request chain. Now, it seems it's actually not that lightweight anymore. Refactoring the business logic is of course a way, a rough way. But adding the raw method back or refining the metadata fetching methods would make our job so much simpler.

[ktalg]

We’re seeing similar allocation pressure in a hot path with FromOutgoingContext, especially when there are multiple AppendToOutgoingContext calls.

I’m thinking of an internal optimization that keeps external semantics the same: replace rawMD.added [][]string with a persistent delta chain:

type rawMD struct {
   md    MD
   added *deltaKV
}
type deltaKV struct {
    parent *deltaKV
    kv     []string
}

Each append would only allocate one node, avoiding the O(n) copy of the prior added list.
If this direction makes sense, I’m happy to take it on. @easwars