Fix some ServerListenerTest tests #4791

Job	Run time
linux-fips-x64	6m 7s
ARM (armv6)	5m 42s
linux-compat (manylinux2014-x86)	6m 5s
linux-compat (manylinux2014-x64)	6m 4s
ARM (arm64)	4m 32s
ARM (armv7)	5m 30s
linux-compat (fedora-34-x64)	6m 8s
linux-compat (al2-x64)	6m 8s
linux-compat (opensuse-leap)	6m 9s
linux-compat (rhel8-x64)	6m 9s
linux-compiler-compat (clang-6)	6m 3s
linux-compiler-compat (clang-9)	6m 5s
linux-compiler-compat (clang-17)	6m 3s
linux-compiler-compat (gcc-5)	6m 4s
linux-compiler-compat (clang-10)	6m 3s
linux-compiler-compat (gcc-4.8)	6m 8s
linux-compiler-compat (clang-8)	6m 8s
linux-compiler-compat (gcc-6)	2m 52s
linux-compiler-compat (clang-11)	6m 7s
linux-compiler-compat (clang-15)	6m 6s
linux-compiler-compat (gcc-11)	6m 8s
windows	6m 8s
linux-compiler-compat (gcc-8)	6m 9s
windows-vc17 (x64)	6m 3s
macos-x64	6m 9s
windows-vc17 (x86)	5m 47s
linux-fips-armv8	5m 28s
localhost-test-win	4m 6s
linux-compiler-compat (gcc-7)	6m 8s
localhost-test-linux	6m 3s
macos	6m 7s
linux-musl-x64 (openwrt-x64-openjdk8)	6m 7s
linux-musl-x64 (alpine-3.16-x64)	6m 7s
localhost-test-macos	6m 9s
check-submodules	1m 2s
GraalVM (macos-latest, 21)	6m 13s
check-docs	58s
GraalVM (windows-latest, 22)	6m 4s
GraalVM (ubuntu-latest, 17)	6m 6s
GraalVM (windows-latest, 17)	6m 2s
GraalVM (macos-latest, 22)	6m 15s
GraalVM (windows-latest, 21)	6m 4s
GraalVM (macos-latest, 17)	6m 15s
GraalVM (ubuntu-latest, 21)	6m 4s
GraalVM (ubuntu-latest, 22)	6m 5s
	4h 16m 0s

Provide feedback