Metrics for consented requests

[maciejkowalczyk]

Hi,
We've noticed that each consented request causes creating a set of metrics with a unique generation_id attribute value.
These metrics accumulate over time until service instance of restarted.

It's possible to filter them in collector service but default otlp collector setup doesn't do it.
This causes unnecessary metrics ingest costs and suggests there is a resource leak in services code

[xinggao01]

Hi Maciej,

We added generation_id to provide metrics per request to help debugging in production. Since the number of consented requests in practice should be low, its impact on production costs should be minimal. Why do you see high volume of consented requests?

To remove it, you can add attributes processor to remove the generation_id in otel collector config.

processors:
  attributes/rm_gen_id:
    actions:
      - key: generation_id
        action: delete
        
service:
  pipelines:
    metrics:
      processors: [batch, attributes/rm_gen_id]  

[fhoering]

It looks like an easy way to kill ad tech traffic as you can just inject a bunch of consented traffic requests with new guids until the servers go down.

On our side we also noticed that there are many metrics with many dimensions that are created and it generates a significant cost already.

[fhoering]

To remove it, you can add attributes processor to remove the generation_id in otel collector config.

It looks like this is possible in GCP but not in AWS as the collector config is part of the AMI and therefore attestation.

[pmeric]

Hi Fabian, the AMI is not attested, only the EIF is attested. Therefore, the AMI can vary, though the EIF must be identical.

[xinggao01]

It looks like an easy way to kill ad tech traffic as you can just inject a bunch of consented traffic requests with new guids until the servers go down.

On our side we also noticed that there are many metrics with many dimensions that are created and it generates a significant cost already.

To send consented requests to B&A servers, it has to include the consented token that is only known to server owner.

Thanks for your feedback. Can you share more details on metrics and dimensions you think are generating unnecessary cost? We will do some investigation.

[maciejkowalczyk]

@xinggao01,

Thanks for the workaround.
IIUC, it "merges" consented requests metrics with normal ones.

We assumed that processing consented requests requires extra computation so they are not interesting from performance POV.
Thus our workaround is to just skip datapoints with generation_id set to sth different than not_consented.

Sure, the volume of consented requests is under our control.
The problem is just the amount of metrics with unique attributes that gets by default created after each consented request and collected even days afterwards.

From our observations it seems that just a single consented request in a full seller+buyer system in GCP generates an additional metric volume of about 50b/s (~$30/month?) which is almost the amount for the 1SFE+1Auction+1BFE+1Bidding system that has never processed a consented request (~60b/s)

[xinggao01]

Are you suggesting that using a static string like generation_id=consented would be preferable to the current generation_id=$unique_id? We can certainly explore that option.

Can you share more about how you get "50b/s" and "$30/month"? A consented request will lead to a one-time increase in metric volume, but I'm unclear why the increase is expressed as a rate (50b/s).

[maciejkowalczyk]

Setting generation_id=consented might possibly be a one way of fixing the issue of services not "forgetting" about consented requests processed long time ago.

In a minimal test B&A seller+buyer cluster that had already processed five consented requests I've applied the following workaround in the collector service config:

processors:
  filter/generation_id:
    error_mode: ignore
    metrics:
      datapoint:
        - attributes["generation_id"] != nil and attributes["generation_id"] != "not_consented"
...
service:
  pipelines:
    metrics:
      processors: [batch, filter/generation_id]  

and observed that monitoring.googleapis.com/billing/bytes_ingested metric has dropped by ~250B/s and 250/5*3600*24*30*0.2580/1e6 is ~30

[xinggao01]

Your filter/generation_id removes metric of consented request.

Do you see 250B/s of metric of consented request even when you are not sending any consented request to server?

[fhoering]

Thanks for your feedback. Can you share more details on metrics and dimensions you think are generating unnecessary cost? We will do some investigation.

@xinggao01 I created a new ticket for this in order not to mix conversations #43
But I also think that the generation_id behavior should be fixed even though AWS billing looks not exactly the same (they don't seem to bill in bytes per hour FWIU)

[maciejkowalczyk]

Your filter/generation_id removes metric of consented request.

Do you see 250B/s of metric of consented request even when you are not sending any consented request to server?

I'm seeing a steady ~70B/s rate of metrics from a 1+1+1+1 system that has not yet received any consented requests.
Each consented request increases this rate by ~50B/s until the workaround is applied.

[xinggao01]

Thanks both. We will work on some design ideas and share here.

[xinggao01]

We will add an option in server flag telemetry_config to configure the metric dimension

it will be able to add or remove these server level dimensions

• deployment.environment
• service.instance.id
• service.name
• service.version
• operator
• region
• zone

and request level dimension

• generation_id: the value can be "not_consented"/ "consent" or request's generation id