Update descriptions for connection timeouts and tcp

[quasiben]

Closes #6636

• Tests added / passed
• Passes pre-commit run --all-files

cc @gjoseph92 @fjetter

Do you think the descriptions should include a note about being potentially useful when resolving deadlocks ?

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

       15 files  ±0         15 suites  ±0   6h 42m 8s ⏱️ + 3m 9s
  2 909 tests ±0    2 823 ✔️  - 1    82 💤 ±0  4 ❌ +2 
21 540 runs  ±0  20 582 ✔️  - 5  953 💤 +3  5 ❌ +3 

For more details on these failures, see this check.

Results for commit c6210a4. ± Comparison against base commit 9b8c3e9.

[fjetter]

Do you think the descriptions should include a note about being potentially useful when resolving deadlocks ?

There is an effect that if the timeouts are configured too large, this could be perceived as a deadlock. Is this what you are referring to?

I think a warning about too large values is appropriate. Something like

Note: If values are chosen too large, a cluster may appear to be stuck if individual workers or the scheduler are waiting for timeouts to expire.

[fjetter]

This is not accurate. This timeout sets a couple of kernel level timeouts that take effect once a connection is established.

Specifically, it sets TCP_USER_TIMEOUT (See https://man7.org/linux/man-pages/man7/tcp.7.html) and configures a KEEPALIVE probe with appropriate intervals. I'm not sure if it is worth it to go into that much detail, though.

The combination of these two settings ensures that a TCP connection is automatically closed if the remote is dead, or rather, the kernel hasn't acknowledged any TCP package in TCP_USER_TIMEOUTs which very likely means the remote is dead.

We use this mechanism, for instance, to infer whether or not a worker died.

[fjetter]

Suggested change

	Timeout after which to error when estabilishing a connection.
	For example, when creating a connection between client and worker,
	client and scheduler, etc.
	All connection attempts are retried until this timeout expires before an
	exception is raised.
	For example, when creating a connection between client and worker,
	client and scheduler, etc.