Initial B&A feasibility and performance assessment

[maciejkowalczyk]

RTB House considers Bidding & Auction (B&A) Services to be a critical component for the potential success of PAAPI. To validate its feasibility, we conducted preliminary load tests on version 4.8.1, aiming to estimate potential costs if B&A were to serve the entirety of retargeting traffic we get from Chrome browser.

Test Setup

• Tested version: 4.8.1

• We reused our existing infrastructure built for on-device PAAPI, including:

  • Production-ready UDF with a <200k-parameter model inference compiled to WASM

  • BYOS key-value service

• We skipped the seller part and sent requests directly to our Buyer Frontend service

• We used simulated, average requests, and searched for maximum stable QPS — defined as a sustained traffic level that produced no errors for at least one hour

Key assumptions

• The ad placement inventory would stay the same after transitioning to PAAPI, including potential multiple requests for the same slot from different SSPs.

• Chaff requests eventually should not significantly affect operational costs.

• About ~25% of incoming real requests are filtered out by the key-value server, and UDF is run on the remaining traffic. Finally, we are placing non-zero bids on ~5% of requests.

• This analysis also did not take k-anonymity into account, i.e. the need for producing multiple bids for each IG

While there is clearly room for optimization, the test provides meaningful signals about feasibility.

Observations

• Using c2d instances on GCP, we observed a throughput of only ~50 IG/s/core, which implies compute costs an order of magnitude higher than our current classical RTB systems.

• NAT costs emerged as an even more significant factor — adding another order of magnitude, largely due to the inflated size of trusted bidding signals, extended due to payload optimization.

Maturity Gaps

We also encountered some early-stage issues beyond performance limitations:

• Missing features (compared to on-device capabilities), e.g. #174, #48, #49, #50

• Bugs surfacing under high load, e.g. #45, #47

• Other issues typical for software in early maturity

Conclusion

RTB House continues to view Bidding & Auction Services as essential for the success of PAAPI. However, our findings suggest that the cost-to-bid-request ratio of B&A in its current state of development is not feasible in the retargeting business.

We believe all these challenges are solvable — but doing so would require meaningful investment from multiple parties across the ecosystem. Unfortunately, we do not observe that right now — which is fully understandable, given the current lack of clarity around long-term plans for third-party cookies and the broader Privacy Sandbox direction.

[fhoering]

Thanks a lot for this ticket that provides valuable insights.

compute costs an order of magnitude higher than our current classical RTB systems

It highlights the importance of a shared and agreed benchmarking methodology to be able to compare results across cloud platforms and to measure the improvements of newly deployed features.

Using c2d instances on GCP, we observed a throughput of only ~50 IG/s/core, which implies compute costs an order of magnitude higher than our current classical RTB systems.

Do you mind sharing in which exact instance you have executed those benchmarks ? Do you use small instances with 1 BFE and 1 bidding server or a different setup ?

On Criteo side we think that it might be better to deploy larger instances in order to mitigate AWS Nitro architecture limitations where only a part of the CPUs can be reserved to the trusted enclave and also to be able to deploy larger ML models with the inference sidecar.
Currently we struggle to get above the 500 QPS limitation but actually think now that it might be a restriction inside the server side B&A components.

NAT costs emerged as an even more significant factor — adding another order of magnitude, largely due to the inflated size of trusted bidding signals, extended due to payload optimization.

Can you confirm you mean egress cost from the BFE to the BYO KV server ? Or do you mean the bidding signals from the IG the would be passed from Chrome to the SFE to the BFE and maybe GCP charges for that ?

[maciejkowalczyk]

Do you mind sharing in which exact instance you have executed those benchmarks ? Do you use small instances with 1 BFE and 1 bidding server or a different setup ?

Yes, we used small instances: a single n2d-standard-2 as BFE and a single c2d-highcpu-4 as Bidding (the actual bottleneck).

NAT costs emerged as an even more significant factor — adding another order of magnitude, largely due to the inflated size of trusted bidding signals, extended due to payload optimization.

Can you confirm you mean egress cost from the BFE to the BYO KV server ? Or do you mean the bidding signals from the IG the would be passed from Chrome to the SFE to the BFE and maybe GCP charges for that ?

In this case, I meant egress from KV server to the BFE. GCP charges for incoming traffic when NAT is used — as it is in default setup). I expect this cost could be avoided with a more customized network setup, though.

[webmihir]

Thanks for sharing these performance data and insights from your B&A service testing. While we test for performance of B&A internally, analyzing specific test setups and results like these is valuable for the ecosystem's ongoing efforts to further optimize B&A services.

To help address the specific performance characteristics you observed, and potentially reproduce the environment to identify optimization opportunities (especially for bidding compute), could you please provide the following details about your specific test setup and results?

1. Request Payload: What was the typical number of Interest Groups (IGs) included per request payload in your simulation? At what QPS were you able to send requests?

2. Throughput: You mentioned a max stable QPS leading to ~50 IG/s/core. As there can be large differences in IG throughput for 1st party cookie based IG vs granular IGs, we suggest using requests/sec/core as a Throughput metric instead of IG/sec/core. However, to help understand this better, could you share the p50 and p95 latencies observed for:- Overall BFE request processing

   • Overall Bidding server request processing
   • UDF execution
   • BYOS KV lookups initiated by B&A

3. Trusted Signal Size: Can you share more details regarding the typical size (e.g., in KB/MB) of the trusted bidding signals being retrieved from the KV server per request or per IG? How does this compare with on-device auctions and have you implemented payload compression with B&A?

Understanding these setup details would allow us to potentially identify hotspots specific to your workload and provide more concrete suggestions for performance improvements.

Regarding network cost, we agree with the recommendation that utilizing more custom network setups, especially with a BYOS KV server, could reduce or potentially eliminate this cost component.

Separately we continue to make improvements aimed at reducing resource consumption and improving throughput for B&A servers. We have an upcoming release 4.9 that will include more optimizations and improvements.

Finally, regarding the other GitHub issues mentioned, we will follow up on the respective threads separately.

Thanks again for initiating this important discussion and sharing your findings. Your answers will help us and the wider community better evaluate the current state.