Many duplicated metrics in AWS

[fhoering]

There seem to be many duplicated metrics exported from the Otel collector to AWS cloud watch which generates a significant cost. Today on a sample deployment with 1 BFE instance and 1 bidding instance I can see over 500 metrics. AWS bills each metric 0.3 USD which means 150 USD cost alone and this doesn't even include the data storage cost and querying cost.

When look at just one metric request.count I get this:

[{'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'telemetry.sdk.language', 'Value': 'cpp'},
   {'Name': 'service.name', 'Value': 'bfe'},
   {'Name': 'service.instance.id', 'Value': 'i-my-instance-id'},
   {'Name': 'service.version', 'Value': '4.7.0'},
   {'Name': 'Noise', 'Value': 'Raw'},
   {'Name': 'generation_id', 'Value': 'not_consented'},
   {'Name': 'telemetry.sdk.version', 'Value': '1.9.1'},
   {'Name': 'region', 'Value': 'us-west-1'},
   {'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'deployment.environment', 'Value': 'prd'},
   {'Name': 'operator', 'Value': 'criteo'},
   {'Name': 'telemetry.sdk.name', 'Value': 'opentelemetry'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'generation_id', 'Value': 'not_consented'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'telemetry.sdk.version', 'Value': '1.9.1'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'Noise', 'Value': 'Raw'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'service.version', 'Value': '4.7.0'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'region', 'Value': 'us-west-1'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'operator', 'Value': 'criteo'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'telemetry.sdk.language', 'Value': 'cpp'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'generation_id', 'Value': 'A-UUID'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'telemetry.sdk.language', 'Value': 'cpp'},
   {'Name': 'service.instance.id', 'Value': 'i-my-instance-id'},
   {'Name': 'service.name', 'Value': 'bfe'},
   {'Name': 'service.version', 'Value': '4.7.0'},
   {'Name': 'generation_id', 'Value': 'A-UUID'},
   {'Name': 'Noise', 'Value': 'Raw'},
   {'Name': 'telemetry.sdk.version', 'Value': '1.9.1'},
   {'Name': 'region', 'Value': 'us-west-1'},
   {'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'operator', 'Value': 'criteo'},
   {'Name': 'deployment.environment', 'Value': 'prd'},
   {'Name': 'telemetry.sdk.name', 'Value': 'opentelemetry'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'deployment.environment', 'Value': 'prd'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'telemetry.sdk.name', 'Value': 'opentelemetry'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'service.name', 'Value': 'bfe'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'service.instance.id',
    'Value': 'i-my-instance-id'},
   {'Name': 'OTelLib', 'Value': 'bfe'}]}]

It seems to be duplicated 15 times and all send the same value except the metrics with {'Name': 'generation_id', 'Value': 'A-UUID'} which corresponds to one consented request that has been sent a long time ago and therefore gets collected as 0.

Most dimensions, except maybe instance_id, look redundant. I already know that I'm criteo, I already know that I'm in us-west-1 as this is a parameter to the CloudWatch query and I already know via the namespace that I'm querying the bfe.
But as they are always the same I would still except to see this metric exactly once with all dimensions and then replicated for different modalities.

So taking into account the current behavior of generation_id (reported here), that I have sent 1 consented request a long time ago, that I have only 1 bfe instance I would expect only 2 lines here one with the actual number of requests that the instance received and one with 0. But it seems not to be the case.

[xinggao01]

Thanks for sharing the details.

What you see the duplicated is how AWS cloud watch organize the dimensions. it is breaking down this

  'Dimensions': [{'Name': 'telemetry.sdk.language', 'Value': 'cpp'},
   {'Name': 'service.name', 'Value': 'bfe'},
   {'Name': 'service.instance.id', 'Value': 'i-my-instance-id'},
   {'Name': 'service.version', 'Value': '4.7.0'},
   {'Name': 'Noise', 'Value': 'Raw'},
   {'Name': 'generation_id', 'Value': 'not_consented'},
   {'Name': 'telemetry.sdk.version', 'Value': '1.9.1'},
   {'Name': 'region', 'Value': 'us-west-1'},
   {'Name': 'OTelLib', 'Value': 'bfe'},
   {'Name': 'deployment.environment', 'Value': 'prd'},
   {'Name': 'operator', 'Value': 'criteo'},
   {'Name': 'telemetry.sdk.name', 'Value': 'opentelemetry'}]},

to

 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'OTelLib', 'Value': 'bfe'}]},
 {'Namespace': 'bfe',
  'MetricName': 'request.count',
  'Dimensions': [{'Name': 'telemetry.sdk.language', 'Value': 'cpp'},

...

Based on aws doc
it treats each unique combination of dimensions as a separate metric.

In your example, there are effectively two separate metrics because the generation_id varies between them. This aligns with the issue discussed in issues/39, where each unique generation_id results in a distinct metric.

Additionally, if more dimensions like instance.id or region are introduced, the number of unique metrics will further increase.

Your suggestion seems to be about reducing unnecessary dimensions that may not be relevant to your use case. We can explore optimizing this by either limiting certain dimensions or making them customizable.

[fhoering]

In your example, there are effectively two separate metrics because the generation_id varies between them.

But when I query CloudWatch I can see 15 metrics for this case. This is not backed by the documentation https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html#Dimension as they really look at the different dimensions values:

Dimensions: Server=Prod, Domain=Frankfurt, Unit: Count, Timestamp: 2016-10-31T12:30:00Z, Value: 105
Dimensions: Server=Beta, Domain=Frankfurt, Unit: Count, Timestamp: 2016-10-31T12:31:00Z, Value: 115
Dimensions: Server=Prod, Domain=Rio,       Unit: Count, Timestamp: 2016-10-31T12:32:00Z, Value: 95
Dimensions: Server=Beta, Domain=Rio,       Unit: Count, Timestamp: 2016-10-31T12:33:00Z, Value: 97

So it would be expected to see 2 metrics but I see 15 and I also see it in the final AWS bill that has a large part of cloud watch on sample instances that don't receive any traffic.

So there seems to be an issue even with the current dimensions and we should also reduce the number of dimensions to make the metrics easier to understand.