gRPC Roundtrip Latency Jitter on LVRT

[jplotzke]

I'm experiencing jitter in roundtrip latency of gRPC calls while running on LabVIEW RT. I've been able to isolate that this jitter is not coming from my specific application (the jitter happens in a very simple example) and would like to understand if this is expected behavior and where this jitter may be coming from.

I have set up a gRPC server with a single server, which has 4 methods to it. Each method does practically nothing, other than returning a current timestamp. I then have set up a parallel loop to the gRPC server which performs a call to these 4 gRPC methods repeatedly. I'm measuring the time of the unary client call while both calling a single method and while calling all 4 methods in parallel. In this fashion, both the server and client are running under LabVIEW RT (on a PXIe-8881 controller) and the client is communicating with the server on localhost (so a physical Ethernet connection is not used).

When iterating over a single gRPC client method call, jitter is reduced, but I still see 'spikes' in the roundtrip call latency. Here is a graph for calling 1 gRPC client call every 10 msec for 2 minutes:

The 'spikes' are exacerbated when calling all 4 methods in parallel. In this graph, the roundtrip time is the time it takes for all 4 methods to return:

I've also tried reducing the interval period of sending these commands. In this graph, I'm sending 4 gRPC client calls in parallel, but at 100 msec intervals (for 20 minutes):

While running on Linux RT and using a localhost connection, I'd expect more deterministic results. Can you help confirm where the latency jitter is coming from?

I have uploaded all of my code that generated these results here: https://github.com/kt-jplotzke/grpc-labview-timingdemo
For reference, I am using the latest grpc-labview release v1.0.0.1.

AB#2342032

AB#2386409

[JamesWiresmith]

I don't have an RT target to hand but it may be worth changing the priority and re-entrancy of the handler VIs and see if that helps. I suspect it won't and the jitter is more likely to be in the network stack but worth a try.

[jplotzke]

I'll admit that I haven't looked deeper into this for several months, but I'm fairly confident that I was able to determine that the network stack was not to blame. I captured some kernel tracing logs while trying to capture the exact point where these "spikes" occurred. My initial findings pointed more towards jitter associated with launching async threads for the grpc client (from this line of code). When the spikes occurred, I saw large memory management functions occurring in the kernel, blocking the async thread from launching for a few usec/msec.

In the end, my main focus was creating a deterministic server (and not a client). When I de-coupled testing the server from the client functions, I was able to run only the gRPC server code and have relatively stable determinism without these spikes. (which makes me further believe that the spikes come from the client functions and not the server functions).

That said, I did not spend much time doing a deep investigation.

[jplotzke]

To add to this issue, I spent some more time recently comparing the different grpc-labview versions that have been released since this issue was first reported, along with different versions of NILRT.

tl;dr: Newer versions of NILRT greatly reduced the measured jitter of a roundtrip gRPC call running both client and server on localhost. I do not believe that grpc-labview had contributed to these initial findings.

Using the same test code as before, I ran this example on NILRT 22.8 (what my initial tests were completed with), 23.8, and 24.8. I kept the same test conditions as before:

• gRPC server and client in the same LabVIEW application, communicating through localhost
• 4 simultaneous client calls being sent every 10 msec
• Measuring execution time to complete all 4 gRPC client unary calls

Under NILRT 22.8, I saw the same 'spikes' in execution time as before:

However, under NILRT 23.8 and 24.8, these spikes were greatly reduced (not completely eliminated, but I'm not surprised there's still a spike here or there). I'm not sure the specifics of what updates may have caused this improvement, but personally I don't care as long as I see the difference and it's repeatable 😝

I ran these tests each with fresh installs of NILRT on two different PXI controllers and obtained the same results each time.

[jplotzke]

Furthermore, I subsequently created packages of this test application for all versions of grpc-labview starting at v1.0.0.1.
Including these results as reference. No issue is being discussed here.

I measured the full execution time for a single client unary call and 4 simultaneous client unary calls 30k times and recorded the mean values below. This is using the same setup as described above -- the gRPC server and client are running in the same LabVIEW application communicating via localhost.

I ran these tests on NILRT 22.8, 23.8, and 24.8. I found no significant difference in mean execution time between NILRT versions.

gprc-labview Version	1 Unary Call Time (usec)	4 Unary Call Time (usec)
1.0.0.1	239	759
1.0.0.2	234	761
1.0.0.3	239	763
1.0.0.4	238	759
1.0.0.5	244	758
1.0.0.6	106	198
1.0.0.7	109	197
1.0.1.1	113	198
1.2.0.1	105	203
1.2.3.1	124	318
1.2.4.1	123	319
1.2.5.1	108	304
1.2.6.1	108	301
1.2.7.1	107	303
1.5.1.1	166	392

Note: v1.2.1.1 and v1.2.2.1 presented issues running on NILRT. I did not investigate and simply did not test these releases. v1.3.0.1+ had an issue (#441) with the NILRT build, so none of these were tested. v1.5.1.1 used the released VIPM packages with the latest build artifact from the main branch with the issue fix implemented

I have highlighted the versions which I found the most change in execution time. Obviously v1.0.0.6 gave the best runtime improvement and there were many changes/improvements in this release.