High latency observed on the listEventGroups call

[madp-acn]

Describe the bug
High latency observed on the listEventGroups call (greater than 10 seconds).

Steps to reproduce
Perform a performance campaign search (this triggers the listEventGroups call).

Component(s) affected
Reporting Server

Version
5.29

Environment
Aquila FE Production

Additional context

[SanjayVas]

We've been discussing this with Arun and Kate. As of now this does not appear to be a Halo bug, so I am closing it for now. It can be reopened if we find evidence that the issue is indeed in Halo software.

1. Everything is working as expected for Origin
2. Latency is fine at the Kingdom and appears to only be high at reporting.

Dashboards shared by Kate. Aquila:

Origin:

[ArunkumarrNatarajan]

@SanjayVas Reopening this issue, adding evidence the response time of the listEventGroup (Mean)

Reporting server- ListEventGroup - Mean response time

Trace of the ListEventGroup

[SanjayVas]

The evidence that's needed is to show that this is a Halo bug rather than something outside of Halo software. In fact, Kate already confirmed that this was an Aquila bug where it was using the deprecated filter field rather than the structured_filter field in the ListEventGroups request. The deprecated field results in the older slower search behavior, meaning it's working as intended from the Halo end.

[SanjayVas]

Ah, I see from the Slack thread that these are being observed even after fixing the Aquila request bug. From Kate:

On Aquila, after further investigations, not all of the high latency listEventGroup queries were caused by using the old filter

We are seeing median latency of around 6 seconds on the listEventGroup API on Aquila when all requests were running the new structured_filter

This looks like a completely separate issue to what we see on Origin. In this case the latency on the Kingdom API is very low and the delay is being introduced by the ReportingServer.

We still need more information to help diagnose. Since the latency is on the Reporting end rather than the Kingdom end, here are some possibilities to investigate:

1. Network latency. e.g. if the Reporting public API server is in a different region or zone than the Kingdom public API server.
2. Resource utilization. e.g. if the Reporting public APIs server is running slower due to high CPU, memory, or thread utilization.