Investigate why server response from the backend can take some time

[jnywong]

Grafana still takes its time waiting for a server response from the backend, even on the order of ~20s at one point:

Originally posted by @jnywong in #53 (comment)

[jnywong]

I don't think the issue is with the fastapi backend, since timing a curl from the Grafana pod to the costs-per-user endpoint took 0.32s for over 15,000 rows of data.

Likely a combination of backend server performance and client side rendering.

[jnywong]

Measuring time to first byte:

kubectl -n support exec -it support-grafana-pod -- curl -w "\nTTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" -s <cost-monitoring-service-ip>/costs-per-user -o /dev/null

TTFB: 3.011109s
Total: 3.011783s

This is an uncached response. Cached responses take this down to

TTFB: 0.229324s
Total: 0.229900s

So we could make some performance optimisations on the backend, but I think the major player is client-side rendering over 15000 rows of data.

[jnywong]

@sunu and I sync'd on this today and changing the Grafana infinity plugin to the Frontend parser appeared to get us down to ~2 seconds, which is on par with a direct call to the backend with port-forwarding.

I will encode this as a change in the Grafonnet code and we can keep an eye on this.

[jnywong]

I think we're okay with this, since for alerting we use Prometheus alertmanager, and for the other features they are not available in the OSS version.

The only loss would be sharing Grafana dashboard snapshots. @sunu how do you feel about that?

[jnywong]

Okay i've deployed on a test cluster and we get down to ~8 seconds on first load and ~4 seconds thereafter. Probably can go a bit further than that

[jnywong]

Next hypothesis: the bottleneck is networking between the Grafana pod and outside traffic.

By process of elimination, we feel the backend is solid, the network between the Grafana and cost-monitoring pod is fine, and the frontend parser is on par with a direct call to the backend with port-forwarding.