Expose HTTP metrics in the Alternator dashboard

[nyh]

A user complained about excessive work being done in TLS handshakes in an Alternator server on HTTPS. We wanted to know if our suspicious that a lot of new Alternator connections are being created per second.
As far as I can tell, we have in the CQL dashboard a rate of new connections per second, but apparently are missing a similar thing in the Alternator dashboard.

We do have the right metric:

scylla_httpd_connections_total{service="API",shard="0"} 0
scylla_httpd_connections_total{service="http-alternator",shard="0"} 0
scylla_httpd_connections_total{service="https-alternator",shard="0"} 0
scylla_httpd_connections_total{service="prometheus",shard="0"} 2

So we can show the rate() of that - the number of new HTTP and HTTPS connections being created each second.

To clarify, this issue is about new connections, not individual requests. But we should also show a count of individual requests if we don't have it yet - this is:

scylla_httpd_requests_served{service="API",shard="0"} 0
scylla_httpd_requests_served{service="http-alternator",shard="0"} 0
scylla_httpd_requests_served{service="https-alternator",shard="0"} 0
scylla_httpd_requests_served{service="prometheus",shard="0"} 2

[amnonh]

@nyh number of new connections is a good metric to have, in CQL we can actually show data on each open connection, and usually, an overgrowing number of connection is an indication for a problem.

I have some doubts about the httpd requests, not against it, but it seems like a low-level metric to show the users. Like showing TCP information in CQL.

[nyh]

@nyh number of new connections is a good metric to have, in CQL we can actually show data on each open connection, and usually, an overgrowing number of connection is an indication for a problem.

I have some doubts about the httpd requests, not against it, but it seems like a low-level metric to show the users. Like showing TCP information in CQL.

Since Alternator is HTTP/HTTPS-based, the scylla_httpd_connections_total{service="https-alternator"} and scylla_httpd_connections_total{service="https-alternator"} metrics are truely the number of new connections, and isn't more "low level" than the "new connections" showed for CQL. I think that if you consider this count to be useful and the "right level" for CQL, it's also useful for Alternator. In some sense what the CQL dashboard of "new connections" is showing is indeed new TCP connections, isn't it?

If in CQL you don't distinguished between new TCP and SSL connections and count both as "new connections" then for Alternator it also makes sense to show the sum of those two metrics scylla_httpd_connections_total{service="https-alternator"} and scylla_httpd_connections_total{service="https-alternator"}.

In general, we have a problem that Alternator and CQL often have a different metric to count the same things. I'm not sure if we consider this a problem, or not. But in the dashboard UI, I think we should usually have similar graphs even if the underlying metric being used to implement them is different.

[amnonh]

@nyh if you look at my comment, I agree with the connection and have doubts about the requests

[nyh]

I misunderstood what you said.
I agree - please add a dashboard graph on derivative of connections (https and http).
The "requests" we already have elsewhere in the dashboard (sum of different labels like PutItem, GetItem, etc., etc.) so indeed we don't need to add another variant of it.