grpc-health-probe shuts down the gRPC connection non-gracefully

[JanDash]

The probe process sends a TCP RST at the conclusion of the gRPC session instead of gracefully shutting down with a FIN/ACK exchange. This causes a lot of noise logging in k8s pods such as:

Feb 01, 2020 2:54:05 PM io.grpc.netty.shaded.io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Screen-shot of Wireshark packet exchange:

[ahmetb]

This came up in #33 as well. However it does not surface up in gRPC implementations other than Java ––so I might need some help debugging this.

The code is rather short, maybe take a look to see if we're missing a conn.Close(). I'll take a look as well.

[ahmetb]

Welp, it seems like we do a bunch of os.Exit(..) which causes defer conn.Close() to be not given a chance to be executed.

Should be a rather easy thing to fix. Would you be able to compile a binary to test if I can give you a patch?

[ahmetb]

See #36, can you compile and let me know if that helps?

[JanDash]

Hi Ahmet, thanks a lot for your quick response! I certainly can, but I need to head out in a minute. I will do it first thing in the morning (PST), is that OK?

[JanDash]

I just built the updated code and tested the binary, but unfortunately I am still getting the same errors.

[ahmetb]

Can you please try this. Change the defer conn.Close() to

defer func(){
    log.Println("closing connection")
    conn.Close()
    log.Println("closed connection")
}()

and recompile & run. We should be seeing these clear messages on the exit path of the probe.

[JanDash]

The messages are printed out but the issue remains. I passed this on to one of my colleagues that knows go to see if he can spot anything.

[ahmetb]

That’s indeed peculiar. Is the wireshark dump the same, or are you seeing RST?

[JanDash]

My coworker made a few changes as an experiment and it still sends RST, same as before.
I am as confused as you are, typically sockets are closed gracefully by the O/S when a process dies. I don't really have any more cycles to experiment further unfortunately, but please feel free to update if you make a breakthrough.

[ahmetb]

My idea would be adding time.Sleep(time.Second*2) before os.Exit() to see if we can keep the process alive for connection to close.

But since you applied the code block above and saw closed connection in the output, honestly I have no other answer.

One good litmus test would be to see if this happens to all Go gRPC clients calling Java gRPC servers. (I assume not, but it might help bisect where the problem is.) For that, you can maybe write a small program that makes an RPC and exits.

[ahmetb]

I went ahead and merged the #36 to master, so you can build off of there.

NB: Issue is still not fixed by #36.

[thejasn]

For issue #36, can we send SIGTERM instead of os.Exit(retCode) ?? Will that solve the issue ??