infinite retry loop READ_LOCAL_CHECKPOINT when a socket is bound to some other process

[ldemailly]

server log:
W0927 21:10:15.014716 1909472 ServerSocket.cpp:103] Error binding :: 59860: Address already in use [98]

I0927 21:10:15.014811 1909472 Receiver.cpp:462] Starting (receiving) server on ports [ 34091 41950 50526 59860 ] Target dir : /data/users/ldemailly_wdt_contbuild/wdtTest/aQP3xB/dst

I0927 21:10:15.115531 1909480 Receiver.cpp:676] Thread[3, port: 59860] entered ACCEPT_WITH_TIMEOUT state

E0927 21:10:17.117660 1909480 Receiver.cpp:696] accept() failed with timeout 2000

I0927 21:10:17.150369 1909476 Receiver.cpp:518] Received done for all threads. Transfer session 1 finished
I0927 21:10:17.150610 1909476 Receiver.cpp:1367] Thread[0, port: 34091] got ack for DONE. Transfer finished
I0927 21:10:17.150753 1909477 Receiver.cpp:1367] Thread[1, port: 41950] got ack for DONE. Transfer finished
I0927 21:10:17.150771 1909478 Receiver.cpp:1367] Thread[2, port: 50526] got ack for DONE. Transfer finished
W0927 21:10:17.150889 1909478 Receiver.cpp:1485] Last thread finished. Duration of the transfer 2.13252
I0927 21:10:17.151083 1909472 Reporting.cpp:240] wdt transfer progress 164.415 Mb, completed 100%, Average throughput 77.0928 Mbps, Recent throughput 0Mbps.
W0927 21:10:17.151116 1909472 Receiver.cpp:308] WDT receiver's transfer has been finished
I0927 21:10:17.151124 1909472 Receiver.cpp:309] Transfer status = OK. Number of blocks transferred = 52. Data Mbytes = 164.415. Header kBytes = 1.33203 (0.000791176% overhead). Total bytes = 172403040. Wasted bytes due to failure = 0 (0% overhead).

client
E0927 21:10:15.027742 1909486 SocketUtils.cpp:124] non-retryable error encountered during socket io 5 0 0: Broken pipe [32]
I0927 21:10:15.027822 1909486 Sender.cpp:723] entered CHECK_FOR_ABORT state 3
I0927 21:10:15.028332 1909486 Sender.cpp:507] Connection took 1 attempt(s) and 0.000262695 seconds. port 59860
I0927 21:10:15.028357 1909486 Sender.cpp:547] entered READ_LOCAL_CHECKPOINT state 3

I0927 21:10:20.031378 1909486 SocketUtils.cpp:150] socket io timed out after 5003 ms, retries 24 fd 5 doneBytes 0
I0927 21:10:20.031636 1909486 Sender.cpp:507] Connection took 1 attempt(s) and 0.000199515 seconds. port 59860
I0927 21:10:20.031651 1909486 Sender.cpp:547] entered READ_LOCAL_CHECKPOINT state 3
I0927 21:10:20.221472 1909487 Reporting.cpp:240] wdt transfer progress 164.415 Mb, completed 100%, Average throughput 31.594 Mbps, Recent throughput 0Mbps.
I0927 21:10:20.421624 1909487 Reporting.cpp:240] wdt transfer progress 164.415 Mb, completed 100%, Average throughput 30.4238 Mbps, Recent throughput 0Mbps.

...

I0928 12:12:17.663367 1909486 SocketUtils.cpp:150] socket io timed out after 5199 ms, retries 25 fd 4 doneBytes 0
I0928 12:12:17.663635 1909486 Sender.cpp:507] Connection took 1 attempt(s) and 0.000202327 seconds. port 59860
I0928 12:12:17.663655 1909486 Sender.cpp:547] entered READ_LOCAL_CHECKPOINT state 3
I0928 12:12:17.674234 1909487 Reporting.cpp:240] wdt transfer progress 164.415 Mb, completed 100%, Average throughput 0.00303785 Mbps, Recent throughput 0Mbps.

[ldemailly]

3 different issues:

a) receiver should bind to both or neither of the ports
b) sender should give up on retries eventually (max retries)
c) sender should know that transfer is done and not even wait for b)