Create api-availability measurement

[mm4tt]

Justification

See this comment - #1086 (comment)

Milestones

V0

1. Create a new "ApiAvailability" measurement
2. This measurement should periodically probe the apiserver's /healthz endpoint and record whether api was available or not
3. The measurement should output the following stats
   1. availability percentage (e.g. % of OK responses over all responses)
   2. Longest consecutively unavailability period
   3. ...

V1

1. Make the measurement fail if apiserver is consecutively not available in the last XX min (e.g. 30min).
2. Make the error critical to make sure the test execution is stopped in such case

V2

1. Come up with exact SLI/SLO definition, make it available in perf-dash for further analysis
2. Make it a WIP Scalability SLO
3. Evaluate and promote to official SLO

/good-first-issue

[k8s-ci-robot]

@mm4tt:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

Details

In response to this:

Justification

See this comment - #1086 (comment)

Milestones

V0

1. Create a new "ApiAvailability" measurement
2. This measurement should periodically probe the apiserver's /healthz endpoint and record whether api was available or not
3. The measurement should output the following stats
   1. availability percentage (e.g. % of OK responses over all responses)
   2. Longest consecutively unavailability period
   3. ...

V1

1. Make the measurement fail if apiserver is consecutively not available in the last XX min (e.g. 30min).
2. Make the error critical to make sure the test execution is stopped in such case

V2

1. Come up with exact SLI/SLO definition, make it available in perf-dash for further analysis
2. Make it a WIP Scalability SLO
3. Evaluate and promote to official SLO

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vamossagar12]

/assign
I would like to take this up. Can I expect more details?

[mm4tt]

Great, thanks @vamossagar12

Let's start with the V0. Could you let me know which part of the description is not clear or requires further details?

[vamossagar12]

thanks @mm4tt Couple of things:

1. how frequently should the probe happen? Would it be configurable or we can define it to be fixed?
2. The availability percentage should be wrt the entire perf test run and we have to keep track of percentage and longest unavailable period?

As an outside question, typically if the health end point is down for a configured time, response would be to scale up or something like that. Just wanted to know the rationale behind adding this particular measurement. Thanks!

[wojtek-t]

how frequently should the probe happen? Would it be configurable or we can define it to be fixed?

Seems like implementation detail. We can make it configurable, but I doubt we will be using different values once we agree on sth.

The availability percentage should be wrt the entire perf test run and we have to keep track of percentage and longest unavailable period?

Correct

As an outside question, typically if the health end point is down for a configured time, response would be to scale up or something like that. Just wanted to know the rationale behind adding this particular measurement. Thanks!

There are couple points:

• we want to ensure that we understand how availability looks like (I can imgine all other SLOs being fine, but cluster being periodicall unavailable)
• we can use it as optimization - if cluster is down for X minutes, the it probably won't get up ever, and we can shut down the test and save money

[vamossagar12]

Thanks for the updates @wojtek-t . I will start with this issue now. Pretty sure there would be a few more questions along the way though :)

[vamossagar12]

hi. Since I last commented, I didn't get a chance to look at it. Will start over the next couple of days..

[vamossagar12]

hi so i started looking at this today. One of the things is that there's already a measurement metrics_for_e2e which, among other things, also fetches the metrics from api-server.

So, the new measurement that needs to be created, can work along the same lines just that instead of hitting the /metrics end-point, it can hit the /healthz end-point of api-server. Thats' from an implementation standpoint.

The other question that I had is that the measurements that we defines are within a Step and a group of Steps forms a single test, so when we say that this new measurement measures the health of the api-server for the duration of a test(mentioned above), so what exactly does a test mean in this case? A Step which houses the measurement(s) or the overall test?

[mm4tt]

Hey, @vamossagar12

It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

Regarding your second question. Usually a measurement has two actions: start and gather. This one should work the same. The start action should start a goroutine that will be continuously pinging /healthz endpoint and the gather will stop this goroutine and wrap everything up. Does it make sense?

[wojtek-t]

It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

Not sure I understand - for metrics_for_e2e IIRC we're fetching the metrics once at the end of test. Assuming that the above is true - it can't really work the same way...

[mm4tt]

It makes sense to make the implementation similar to the metrics_for_e2e. It's exactly as you said, instead of hitting /metrics, we'll be hitting /healthz.

Not sure I understand - for metrics_for_e2e IIRC we're fetching the metrics once at the end of test. Assuming that the above is true - it can't really work the same way...

Good point, I should have checked how the metrics_for_e2e works :) Still you should be able to take some inspiration from that measurement. Let me know if you have more questions.

[vamossagar12]

Actually I meant to use the metrics_e2e as only a baseline of how to interact with the api server. I hadn't looked at the internals so what @wojtek-t told is even more valuable :)