Indefinite blocking and unable to receive data

[kazikame]

Expected Behavior

TAS should send/receive packets without waiting indefinitely

Current Behavior

TAS sometimes fails to send (or receive) data sent by the last call to send(). This causes the receiver to wait indefinitely even after the sender has stopped.

Steps to Reproduce

The bug is non-deterministic and may happen at the server or the client, however it can be reproduced fairly reliably using the following server and client in this repo

1. Compile using -Ofast and -march=native
2. Run TAS on both the server and client
3. Server:

   LD_PRELOAD=<path-to-libtas_interpose.so>  ./server <server-ip> <server-port

4. Client:

   LD_PRELOAD=<path-to-libtas_interpose.so>  ./client<server-ip> <server-port

Context (Environment)

The bug was discovered when this software RDMA stack was attempted over TAS on two machines equipped with the 10G Intel 82599 NICs. All performance benchmarks get blocked indefinitely on TAS. They run fine on the regular kernel TCP stack.

[PabstMatthew]

One thing we've also noticed about this issue is that decreasing --fp-poll-interval-tas seems to make the success probability decrease.

[rajathshashidhara]

TAS is designed for long running applications -- it has an implicit assumption that applications indefinitely poll the network stack for updates. The sample code you've shared makes a fixed number of socket calls. Due to this, the client fails to propagate the last transmission update to its fastpath causing the server to infinitely block on recv(). This liveness condition is easy to satisfy -- ensure that your code polls on flextcp_context_poll() indefinitely. Note that the sockets API layer in TAS relies on this function to propagate updates to the fastpath.