Concurrent map read and write in span_concentrator.go

[ax13090]

Hello Datadog developers,

My company's Go code is using dd-trace-go to send APM Spans to Datadog. We occasionally see error logs mentioning a concurrent map read and write, and I wanted to report this here. The following goroutine trace seems relevant.

fatal error: concurrent map read and map write
goroutine 57743428 [running]:
internal/runtime/maps.fatal({0x1c1e077?, 0xc00a8dc978?})
/usr/local/go/src/runtime/panic.go:1046 +0x18
github.com/DataDog/datadog-agent/pkg/trace/stats.matchingPeerTags(0xc001f192f0, {0xc000148908, 0x2b, 0x47})
/go/pkg/mod/github.com/!data!dog/datadog-agent/pkg/trace@v0.71.2/stats/span_concentrator.go:56 +0x147
github.com/DataDog/datadog-agent/pkg/trace/stats.(*SpanConcentrator).NewStatSpanWithConfig(0xc000900210, {{0x1bf0e3c, 0xb}, {0x1c2764d, 0x25}, {0x1bf0e3c, 0xb}, {0x1be6c97, 0x3}, 0x53ea5b5cb71ff2ab, ...})
/go/pkg/mod/github.com/!data!dog/datadog-agent/pkg/trace@v0.71.2/stats/span_concentrator.go:183 +0x1ee
github.com/DataDog/datadog-agent/pkg/trace/stats.(*SpanConcentrator).NewStatSpan(0x20?, {0x1bf0e3c?, 0x1717760?}, {0x1c2764d?, 0x100000101?}, {0x1bf0e3c?, 0x0?}, {0x1be6c97?, 0xc0001eedda?}, 0x53ea5b5cb71ff2ab, ...)
/go/pkg/mod/github.com/!data!dog/datadog-agent/pkg/trace@v0.71.2/stats/span_concentrator.go:206 +0x178
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer.(*concentrator).newTracerStatSpan(0xc000476340?, 0xc0066ce6c0, 0xc00f2cd7e0?)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.3.0/ddtrace/tracer/stats.go:169 +0x14c
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer.(*tracer).submit(0xc00037e460, 0xc0066ce6c0)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.3.0/ddtrace/tracer/tracer.go:964 +0x72
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer.(*Span).finish(0xc0066ce6c0, 0x18741a794ed545c3)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.3.0/ddtrace/tracer/span.go:765 +0x377
github.com/DataDog/dd-trace-go/v2/ddtrace/tracer.(*Span).Finish(0xc0066ce6c0, {0x0, 0x0, 0x1772dc0?})
/go/pkg/mod/github.com/!data!dog/dd-trace-go/v2@v2.3.0/ddtrace/tracer/span.go:689 +0x28e
github.com/DataDog/dd-trace-go/contrib/google.golang.org/grpc/v2.finishWithError(0xc0066ce6c0, {0x1e4dba0, 0x2eba0f0}, 0xc0005a4620)
/go/pkg/mod/github.com/!data!dog/dd-trace-go/contrib/google.golang.org/grpc/v2@v2.3.0/grpc.go:109 +0x231
github.com/DataDog/dd-trace-go/contrib/google.golang.org/grpc/v2.StreamClientInterceptor.func1.2()
/go/pkg/mod/github.com/!data!dog/dd-trace-go/contrib/google.golang.org/grpc/v2@v2.3.0/client.go:116 +0x77
created by github.com/DataDog/dd-trace-go/contrib/google.golang.org/grpc/v2.StreamClientInterceptor.func1 in goroutine 57737262
/go/pkg/mod/github.com/!data!dog/dd-trace-go/contrib/google.golang.org/grpc/v2@v2.3.0/client.go:114 +0x308

This is Go 1.24.2. With datadog-agent version 0.71.2, apparently an indirect dependency of dd-trace-go version 2.3.0.

This is my first report here, so feel free to request more information that I didn't include here.
Thanks!

[anais-raison]

Hello @ax13090 !
Could you send us the rest of the stack trace, if there is one, so we can further investigate the source of the error?
Thanks.

[AndreySlashman]

Maybe this will help @anais-raison

Nov 7, 2025 @ 10:44:12.560 | created by vidoq/vendor/gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer.initalizeDynamicInstrumentationRemoteConfigState in goroutine 1 | vidoq
-- | -- | --

  | Nov 7, 2025 @ 10:44:12.560 | vidoq/vendor/gopkg.in/DataDog/dd-trace-go.v1/internal/remoteconfig.Start.func1.1() | vidoq

  | Nov 7, 2025 @ 10:44:12.560 | - | vidoq

  | Nov 7, 2025 @ 10:44:12.560 | vidoq/vendor/github.com/DataDog/datadog-go/v5/statsd.(*sender).sendLoop(0xc0004a0000) | vidoq

  | Nov 7, 2025 @ 10:44:12.560 | created by vidoq/vendor/github.com/DataDog/datadog-go/v5/statsd.newSender in goroutine 1 | vidoq

  | Nov 7, 2025 @ 10:44:12.557 | fatal error: concurrent map read and map write | vidoq

[anais-raison]

Hello @AndreySlashman !
We're still having some difficulty identifying the root cause here. The span concentrator code expects that it will have exclusive access to the fields of a span, which is protected by a span mutex that is locked during the span.Finish call. Our current hypothesis is that there is some goroutine accessing the internals of the span without using a lock, but without the full stacktrace we can't see where. It looks like you've redacted some data from your stacktrace which is understandable given this is a public issue tracker. If you would feel more comfortable, you can open a support issue with Datadog and share the full stacktrace there that would greatly assist in our investigation.