[FEATURE] Filter out resources from cloudwatch where null

[THOM-AwS]

Is there an existing issue for this?

• I have searched the existing issues

Feature description

Our organization has some accounts that have a history of a lot of ec2 running or having been cycled on the account, numbering in the hundreds. It appears that YACE queries is asking each ec2 for metric data, and that is creating a huge bill for those accounts. Is there any feature we can get to filter out metrics in cloudwatch that only contain null values, and save on getmetricdata costs?

What might the configuration look like?

apiVersion: v1alpha1
  sts-region: ap-southeast-2
  discovery:
    exportedTagsOnMetrics:
      ec2:
        - Environment
    jobs:
    - type: ec2
      delay: 120
      roles:
        - roleArn: "arn:aws:iam::1234:role/GrafanaRole"
        - roleArn: "arn:aws:iam::5678:role/GrafanaRole"
        - roleArn: "arn:aws:iam::9101112:role/GrafanaRole"
      regions:
        - ap-southeast-2
      statistics:
        - Minimum
        - Maximum
        - Sum
        - Average
      metrics:
        - name: CPUUtilization
          period: 60
          length: 300
          nilToZero: true
        - name: DiskReadOps
        - ETC

Anything else?

No response

[kgeckhart]

I'm unsure how this is possible because the only way for the exporer to know the metric is null is to call GetMetricData.

There's a couple things which are already in place to help with this though,

1. Only tagged resources are returned and deleted instances are quickly removed from the Tagging API reducing calls to GetMetricData. This doesn't reduce calls to ListMetrics.
2. You can add recentlyActiveOnly to your job, https://github.com/nerdswords/yet-another-cloudwatch-exporter/blob/master/docs/configuration.md#discovery_job_config will limit resources returned from ListMetrics being only those which have produced a data point to CloudWatch in the last ~3 hours.
3. Consider using tag filters to further reduces the data being gathered

Do you have more ideas on how this might be possible?

[THOM-AwS]

I'm unsure how this is possible because the only way for the exporer to know the metric is null is to call GetMetricData.

There's a couple things which are already in place to help with this though,

1. Only tagged resources are returned and deleted instances are quickly removed from the Tagging API reducing calls to `GetMetricData`. This doesn't reduce calls to  `ListMetrics`.

2. You can add `recentlyActiveOnly` to your job, https://github.com/nerdswords/yet-another-cloudwatch-exporter/blob/master/docs/configuration.md#discovery_job_config will limit resources returned from `ListMetrics` being only those which have produced a data point to CloudWatch in the last ~3 hours.

3. Consider using tag filters to further reduces the data being gathered

Do you have more ideas on how this might be possible?

@kgeckhart Thanks a lot for the response, It was not until shortly after posting that I came across the recentlyActiveOnly feature flag. I plan on trying to implement this soon.

Unfortunately the cloudwatch service does not support tagging in this way afaik, and the important metrics we want to collect for our monitoring is in the cwagent custom namespace. I was hoping there would be a bit more control over filtering somehow, other than just the three hour activity, but this may end up being the thing we need.

Tagging at our scale is quite difficult, as we have hundreds of accounts and thousands of ec2, that are not easy to update process wise.

Im aware that when making api calls with awscli you can add a filter to the query and only those things included are returned and charged for. I take it thats not an option here.