Open
Description
Describe the bug
Docker 25.0.6 service crashes intermittently when Gelf log driver is used
To Reproduce
- Start an instance using AWS ECS AMI version 2023.0.20250115
- Configure a logstash server to accept Gelf inputs
- Configure an ECS task to use Gelf and send logs to Logstash
Expected behavior
Docker service shouldn't crash.
Screenshots
Logs from journalctl -u docker.service:
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: SIGSEGV: segmentation violation
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: PC=0x557343aaf097 m=222 sigcode=1 addr=0x20
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: goroutine 0 gp=0xc0023a8000 m=222 mp=0xc001c15808 [idle]:
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.(*mspan).typePointersOfUnchecked(0xc00290d000?, 0xc00207a450?)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mbitmap_allocheaders.go:202 +0x37 fp=0x7fd31f5fdc10 sp=0x7fd31f5fdbf0 pc=0x557343aaf097
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.scanobject(0xc00006b768?, 0xc00006b768)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mgcmark.go:1441 +0x1ce fp=0x7fd31f5fdca0 sp=0x7fd31f5fdc10 pc=0x557343abbb8e
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.gcDrainN(0xc00006b768, 0xf4d44)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mgcmark.go:1331 +0x17a fp=0x7fd31f5fdcd0 sp=0x7fd31f5fdca0 pc=0x557343abb79a
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.gcAssistAlloc1(0xc0024b7880, 0xf4d44)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mgcmark.go:665 +0x114 fp=0x7fd31f5fdd30 sp=0x7fd31f5fdcd0 pc=0x557343aba2b4
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.gcAssistAlloc.func1()
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mgcmark.go:556 +0x1b fp=0x7fd31f5fdd50 sp=0x7fd31f5fdd30 pc=0x557343aba17b
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.systemstack(0xa00000)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/asm_amd64.s:509 +0x47 fp=0x7fd31f5fdd60 sp=0x7fd31f5fdd50 pc=0x557343b103c7
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: goroutine 5893 gp=0xc0024b7880 m=222 mp=0xc001c15808 [GC assist marking]:
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.systemstack_switch()
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/asm_amd64.s:474 +0x8 fp=0xc0006af958 sp=0xc0006af948 pc=0x557343b10368
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.gcAssistAlloc(0xc0024b7880)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/mgcmark.go:555 +0x205 fp=0xc0006af9b8 sp=0xc0006af958 pc=0x557343ab9ee5
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.deductAssistCredit(0xc0006afa68?)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/malloc.go:1353 +0x54 fp=0xc0006af9e0 sp=0xc0006af9b8 pc=0x557343aa75d4
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.mallocgc(0xa0508, 0x557346074620, 0x1)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/malloc.go:1025 +0xcc fp=0xc0006afa68 sp=0xc0006af9e0 pc=0x557343aa6d4c
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: runtime.newobject(0x0?)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/runtime/malloc.go:1390 +0x25 fp=0xc0006afa90 sp=0xc0006afa68 pc=0x557343aa76c5
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: compress/flate.NewWriter(...)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/compress/flate/deflate.go:666
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: compress/gzip.(*Writer).Write(0xc00154dad0, {0xc000453500, 0x303, 0x1500})
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /usr/lib/golang/src/compress/gzip/gzip.go:191 +0x2cd fp=0xc0006afaf0 sp=0xc0006afa90 pc=0x557343ebd32d
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: github.com/Graylog2/go-gelf/gelf.(*UDPWriter).WriteMessage(0xc0019d35c0, 0xc001887380)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /builddir/build/BUILD/docker-25.0.6-1.amzn2023.0.2/src/github.com/docker/docker/vendor/github.com/Graylog2/go-gelf/gelf/udpwriter.go:196 +0x37f fp=0xc0006afca0 sp=0xc0006afaf0 pc=0x55734553cebf
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: github.com/docker/docker/daemon/logger/gelf.(*gelfLogger).Log(0xc0020ca500, 0xc000ccc3f0)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /builddir/build/BUILD/docker-25.0.6-1.amzn2023.0.2/src/github.com/docker/docker/daemon/logger/gelf/gelf.go:185 +0x2c2 fp=0xc0006afd90 sp=0xc0006afca0 pc=0x55734553eac2
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: github.com/docker/docker/daemon/logger/loggerutils/cache.(*loggerWithCache).Log(0xc0019dd300, 0xc000ccc3f0)
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /builddir/build/BUILD/docker-25.0.6-1.amzn2023.0.2/src/github.com/docker/docker/daemon/logger/loggerutils/cache/local_cache.go:78 +0x1ce fp=0xc0006afdf0 sp=0xc0006afd90 pc=0x55734474012e
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: github.com/docker/docker/daemon/logger.(*Copier).copySrc(0xc001feb680, {0x5573457d61e4, 0x6}, {0x7fd3457b2260, 0xc0010cb400})
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /builddir/build/BUILD/docker-25.0.6-1.amzn2023.0.2/src/github.com/docker/docker/daemon/logger/copier.go:129 +0xe06 fp=0xc0006affa8 sp=0xc0006afdf0 pc=0x55734472dc06
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: github.com/docker/docker/daemon/logger.(*Copier).Run.gowrap1()
Jan 27 23:00:10 ip-??-?-??-???.ec2.internal dockerd[2614]: /builddir/build/BUILD/docker-25.0.6-1.amzn2023.0.2/src/github.com/docker/docker/daemon/logger/copier.go:49 +0x30 fp=0xc0006affe0 sp=0xc0006affa8 pc=0x55734472cdd0
...