Flaky in in_process_relay_test: "connection refused"

[drigz]

I observed the following flake:

$ bazel test //src/go/tests/relay:in_process_relay_test --runs_per_test=100 --test_output=errors
[...]
2025/03/20 15:22:04 ERROR BackendRequest ID=server_name:6ec6703aa3b52e4291e4ae3fbfa87bbc Message="Backend request failed with error: Get \"http://127.0.0.1:34391/\": dial tcp 127.0.0.1:34391: connect: connection refused"
[...]
--- FAIL: TestHttpErrorPropagation (0.04s)
    --- FAIL: TestHttpErrorPropagation/Propagate_http.StatusHTTPVersionNotSupported (0.00s)
        in_process_relay_test.go:195: Server responeded with an unexpected status code.
                Expected: 505
                Observed: 500
[...]
//src/go/tests/relay:in_process_relay_test                               FAILED in 3 out of 100 in 1.4s

I'll add flaky=true to mitigate this.

[koonpeng]

Here are my investigations

core/src/go/tests/relay/in_process_relay_test.go

Lines 105 to 107 in acee658

	go func() {
	srv.ListenAndServe()
	}()

It is possible for the request to come in before the backend is ready.

core/src/go/tests/relay/in_process_relay_test.go

Lines 27 to 39 in acee658

	func pickUnusedPortOrDie() int {
	var addr *net.TCPAddr
	var err error
	if addr, err = net.ResolveTCPAddr("tcp", "localhost:0"); err == nil {
	var list *net.TCPListener
	if list, err = net.ListenTCP("tcp", addr); err == nil {
	defer list.Close()
	return list.Addr().(*net.TCPAddr).Port
	}
	}
	glog.Fatal("Failed to pick a free TCP port.")
	return 0
	}

The tests listens on port 0, letting the kernel pick a free port, immediately close the listener, then use the same port to start the server. But closing the listener does not immediately free up the port, it is possible for the address to still be bounded when starting the server.

Some ideas to fix:

• Use unix sockets, but I don't think the current code supports it.
• Allow to the relay server with a bounded listener, this is not currently supported as well.
• Retry until success, does not require changes in the code, but is more hacky and makes the test slower.