httpd should close connections that have been idle for a long time

[nyh]

#3129 reports that: "What we are seeing with AWS load balancers is that they silently drop idle connection state after a while. This is transparent to the server, the connections just stays established forever.".

For this and many other reasons (clients disappearing, rebooting, etc.) our http server might end up with a very large number of idle connection, supposedly kept alive for the client to reuse them but the client will never do and these connections will never get closed. This is bad, and even dangerous.

In #3129, it was suggested to use kernel-level TCP keepalive (with low settings) to solve this problem. But #3129 did not turn this feature on by default - it just provided a mechanism. I'm suggesting here that perhaps we should turn this feature on by default.

Alternatively, instead of using TCP keepalive, we could do something even simpler in user-space - we could automatically close any idle connection which is older than some time limit - not even checking if the client is still alive. For example, Apache HTTPd has KeepAliveTimeout https://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout - which defaults to just 5 seconds (!). Nginx has keepalive_timeout https://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout which defaults to 75 seconds. Every other HTTP server I looked at seems to have such a timeout, and its default configuration is usually just a few seconds. I don't know if Seastar's default should be so low, but infinity is too high :-)

[nyh]

In #3129 @avikivity said he prefers the timeout approach (what I called above "alternatively"), not the TCP keepalive:

httpd should have the ability to garbage-collect idle connections.
This puts the keep-alive onus on the client, where it should be.
Suppose we replaced this with a per-connection timer that closes the connection after N minutes if idleness: wouldn't it be better? avoid the (small amount of) work in sending keepalives.

[nyh]

Although much of the discussion revolved around HTTP keepalive and the parameters of Apache and Nginx quoted above use this word in their name, and the most common reason why we need to close old connections has to do with keepalive, on further thinking I realized the problem is not relevant only to keepalive:

A client may disappear (e.g., reboot) while in the middle of sending a request to our server. In that case, the server will continue waiting for the rest of the request - forever. Unless we either enable TCP keepalive (#3129) or we close connections that are hung in the request read for a long time (this issue).

So we should close a connection if it's hung on a read for a long time - regardless of whether we're just beginning to read the request, or we're in the middle of the read.

Note that to support huge requests over low bandwidth, we to allow the entire request to span a long time - we should timeout just individual reads.