[FEATURE] New metrics to track Cloudwatch API requests and response errors

[rtkwlf-nachiketrao]

Is there an existing issue for this?

• I have searched the existing issues

Feature description

Feature request:

Metric to count the number of requests made to cloudwatch API (irrespective of response) and count request errors if no valid data is returned.
OR
Include such a metric as part of yace_cloudwatch_requests_total and yace_cloudwatch_request_errors.

Reason:

No way to detect case where the IAM authentication fails or Cloudformation stack is misconfigured. A request is made, but there is no response. There seems to be no counter for this.

How this became an issue:

While testing for correctness of YACE Cloudwatch API requests using yace_cloudwatch_requests_total and yace_cloudwatch_request_errors:

• Intentionally removed IAM from service account for YACE pod
• service was denied, but yace_cloudwatch_requests_total doesn't increment - seems like it counts the number of "successful responses to requests" and not "number of requests irrespective of responses received".

Having metrics to count total requests submitted and response errors will help detect such edge cases.

What might the configuration look like?

No response

Anything else?

https://github.com/nerdswords/yet-another-cloudwatch-exporter/blob/edd637e19e7eaa889bd75ab24408549f38eaf0ab/pkg/clients/cloudwatch/v1/client.go#L97

^ similar counter, but should be incremented before a request is made to the API. and counter to check for request errors

[kgeckhart]

👋 #1338 isn't exactly what you are asking for but I think solves the problem you are having. Every CloudWatch API call should now either increment the success or error metric depending on the result.

Auth errors should also be present in the logs as an indicator something is wrong.