Random Code Stuck in gRPC Client Unary Call

[kthirunx]

Hi Team,

I am facing a code stuck issue in gRPC Client Unary Call, as seen below.

Setup:

1. Sample gRPC Server/Client generated from gRPC 1.6.1.1 in LabVIEW 2020 64-bit.
2. LabVIEW gRPC Server built as an executable
3. LabVIEW gRPC Client Example running from LabVIEW

What the Client Example Does:

1. Launch server EXE
2. Retry sending a message to server till successful, max 5 times.
3. When the server receives the message, it will stop and exit.
4. Repeat steps 1 to 3 for 1000 trials. In this case, the code got stuck in the 327th trial.

Other Observations:

1. Server EXE is still running when the stuck issue is seen in client.
2. From another LabVIEW version (2019 64-bit), I was able to send a message and get response from server EXE. This likely indicates that the server is functional, and the issue seems to be on the LabVIEW gRPC client side.
3. Pressing the abort button in LabVIEW doesn't work and I had to use 'End Task' from Task Manager.

Versions Used:

LabVIEW gPRC Version: 1.6.1.1
LabVIEW 2020 64-bit

Sample App (LabVIEW 2020 64-bit):

LV2020_gRPC_v1.6.1.1.zip

[yash-ni]

Try toggling useOccurrence . Refer to this to see how to enable or disable features.

[jasonmreding]

FWIW, your client example that attempts to send the message five times until successful is using an infinite timeout each time with no delay between retries. So if the attempt doesn't immediately return an error (likely because the server process hasn't spun up yet), then it's going to wait forever. Inserting a delay after the failed attempt might also be helpful if the process takes longer than expected to start up. I suspect if you used a non infinite timeout value on the order of a few seconds, it will likely work around the issue. That's not to say there isn't an underlying bug that might need to be investigated still, but I suspect this approach will be more robust in general if your client needs to handle the situation where the service might not be running yet. This is what similar logic looks like in one of our test suites.

[kthirunx]

Thanks team for the suggestions, we will check on this and get back.

[jasonmreding]

After poking at this a bit today, I'm pretty sure the issue is in your test setup. Your client program is instructing the service to stop/exit with each RPC it sends. It then immediately tries to restart the service on the next execution of the loop again. This inherently creates a race condition between the shutdown of the service launched from the previous loop iteration and the service being started from the current loop iteration. If the service instance from the previous loop iteration completely shuts down before the new service instance is launched from the current loop iteration, everything works fine. However, if there is any overlap between the lifetime of the two service instances, then you are going to run into trouble:

• Since you are using a static port assignment, the current service instance may fail to start because the port is still in use by the previous service instance.
• The client request for the current loop iteration might get routed to the previous loop iteration's service instance which is shutting down. This will generally result in error -1014 which indicates the service endpoint is UNAVAILABLE.

You can also run into TCP/IP socket exhaustion issues due to TIME_WAIT state with this methodology depending on the configuration of your network stack. Bottom line is you either need to extend the lifetime of your server or you need to perform some other handshaking with your client application if you want this sort of thing to work reliably. For example, the client could monitor and wait for the service process to exit after telling it to exit using standard Windows APIs. You could also consider assigning a dynamic port during startup of the service. However, that means you will need to come up with some other IPC mechanism to communicate the dynamically assigned port with your clients.

[kthirunx]

Thanks Jason, this is helpful to know.

The stuck issue faced in the above example could likely be due to message getting routed to a service shutting down and the client ends up getting stuck/waiting indefinitely for a response.

To make the code more robust, would it be possible for the client to generate an error if the service shuts down, even if the timeout is -1, as there is no way this request is going to get completed?

[jasonmreding]

To make the code more robust, would it be possible for the client to generate an error if the service shuts down, even if the timeout is -1, as there is no way this request is going to get completed?

That is my expectation, and it is what I am seeing when trying to reproduce the issue as you described above. If I don't modify your attached example code at all, then I am rarely able to see more than one or two successful iterations before the client stops with a -1014 error. If I add a one second retry delay and allow for up to 10 retries before stopping (still using infinite timeout with each RPC), then I usually see tens of successful runs of the loop before it again errors out with a -1014 error. In these cases, the exe is never running when the client loop stops.

I then added a modal dialog to the server if it fails to start since I suspected I was running into a duplicate port errors on startup of the second process. However, I never saw the modal dialog pop up even after the client loop ended in an error. I then realized that LV executables don't allow multiple instances by default so I added allowmultipleinstances = TRUE to the ini file. With this configuration, the loop runs much faster, and the client ran a full 1000 iterations for me without error. Did you also modify your ini file in a similar fashion? If not, the behavior I am observing is considerably different from what you described.