docs(WIP): Update AWS Guide with notes on Proxy and Perf

Discusses how the root of the differences between AWS and GCP is the difference between AWS Nitro Enclaves and GCP Confidential Compute Spaces, how this requires the AWS TCP-to-VSOCK proxy, and that (and how) this affects server performance.
Also updates the self-serve guide to include our scaling recommendations
for each service, and AWS-specific enclave CPU allocation
recommendations.

Author: akundla-google

bidding_auction_services_aws_guide.md

@@ -18,11 +18,21 @@ The FLEDGE Bidding and Auction services will be open-sourced in Q2 2023. In addi
 
 
 #### Comparison to Google Cloud Platform (GCP)
-While this document focuses on AWS, [a similar document was published for GCP in Q2 2023][53]. We intend to provide full GCP and AWS support for these services. Most implementation details will be, at a high level, similar. A few notable exceptions (ultimately with no impact on functionality) are:
+While this document focuses on AWS, [a similar document was published for GCP in Q2 2023][53]. We intend to provide full GCP and AWS support for these services. The API for each platform is identical. Most implementation details will be, at a high level, similar.
+
+However, AWS's Nitro Enclaves differ significantly from GCP's confidential compute spaces architecturally. [GCP confidential spaces share the same kernel as their host machine and permit most network communication natively][5], whereas AWS Nitro Enclaves run a fully-separate linux kernel and can communicate with the host OS only over the VSOCK protocol. As a result of these additional complications introduced by the AWS approach to TEEs:
+* AWS instances run a TCP-to-VSOCK proxy to enable network ingress and egress to and from the enclave.
+* This proxy has elements running both inside and outside the enclave, and compute resources (vCPU cores) must be allocated accordingly.
+* Due to both the need for a proxy and its limitations, networking performance characteristics of B&A servers on AWS are not identical to GCP. Specifically, in our testing:
+  * Instances of the backend services (auction service and bidding service) perform very similarly to GCP in terms of queries per second per vCPU.
+  * Instances of the frontend services (Buyer Front-End Service and Seller Front-End Service) may not be able to serve the same throughput as equivantly-sized instances on GCP. This is dependent on the amount of network traffic performed in the key-value service lookup.
+
+We highly recommend AdTechs refer to [both the general and  AWS-specific notes on service scaling behavior in the self-serve guide][56]. These provide crucial guidance for determining number and size of instances on AWS and allocating memory and vCPU cores between the parent and enclave (a concept particular to AWS TEEs).
+
+A few other notable exceptions (ultimately with no impact on functionality):
 
-1. GCP is based on a single [virtual machine instance][5], so a parent-child instance relationship will not be required to host the running trusted code in GCP.
 1. The [Envoy component][4] runs inside the Seller Frontend TEE in GCP.
-1. Each EC2 instance in AWS runs the following outside TEE: a health-checking script, and an instance of Envoy in a container. These support the [service mesh][3] for AWS. No such infrastructure is needed for mesh on GCP.
+1. When deployed and run with [service mesh][3], each EC2 instance in AWS runs the following outside TEE: a health-checking script, and an instance of Envoy in a container. No such infrastructure is needed for mesh on GCP.
 
 ### Sell-side platform
 
@@ -368,3 +378,4 @@ grpcurl -d '@' dns:///<DOMAIN.COM>:443 privacy_sandbox.bidding_auction_servers.<
 [53]: https://github.com/privacysandbox/bidding-auction-servers/tree/main/production/deploy/aws/terraform/environment/demo
 [54]: https://aws.amazon.com/blogs/containers/migrating-from-aws-app-mesh-to-amazon-ecs-service-connect/
 [55]: https://github.com/WICG/protected-auction-services-discussion/issues/
+[56]: https://github.com/privacysandbox/fledge-docs/blob/main/bidding_auction_services_onboarding_self_serve_guide.md#estimate-an-instance-size

bidding_auction_services_onboarding_self_serve_guide.md

@@ -684,8 +684,8 @@ The following parameters are optional, per cloud region:
 The following scaling and latency strategy will help determine the size and
 number of instances required to serve scaled traffic with an acceptable latency.
 This would involve the following:
- * Vertical scaling
- * Horizontal scaling
+ * Vertical scaling (use a larger instance for the service)
+ * Horizontal scaling (add more instances of the same size for the service)
 
 #### Vertical scaling
 
@@ -801,6 +801,16 @@ number of ads per request, and the average size of `trustedScoringSignals` per a
   </tr>
 </table>
 
+Seller Front-End Service Scaling Properties:
+
+For many workloads with which the B&A team has tested, especially those with large KV signal fetches, SFE Service is neither CPU- nor memory- bound; that is, it needs enough of each of these resources but will hit its maximum throughput without exhausting either.
+
+This effect is much more pronounced on AWS than on GCP, due to the AWS TCP-to-VSOCK proxy, but exists on both platforms to some degree due to service architecture. 
+
+As a result, SFE tends to scale poorly vertically, but scales well horizontally. If you have exhausted the throughput of your SFE, we recommend you add another SFE of a similar size, and so on, to increase throughput and decrease latency. 
+
+On AWS, we recommend AdTechs start their dialing-in by reserving _half_ of the instance's vCPU cores and 4GB memory for running the parent, regardless of instance size. 
+
 ###### Auction Service
 
 The Auction service usually requires higher compute power than the SFE to support
@@ -827,6 +837,12 @@ the same RPS, since every `scoreAd()` execution happens in isolation per ad.
   </tr>
 </table>
 
+Auction Service Scaling Properties:
+
+For many workloads with which the B&A team has tested, Auction Service is CPU-bound; that is, its throughput is scales primarily with available compute power. Auction service thus scales well both vertically and horizontally. 
+
+On AWS, we recommend AdTechs start their dialing-in by reserving 2 vCPU cores and 4GB memory for running the parent, regardless of instance size. 
+
 ###### Determine SFE / Auction instances ratio
 
 A **single** SFE instance can support a higher RPS with multiple Auction server
@@ -877,6 +893,16 @@ per interest group.
   </tr>
 </table>
 
+Buyer Front-End Service Scaling Properties:
+
+For many workloads with which the B&A team has tested, especially those with large KV signal fetches, BFE Service is neither CPU- nor memory- bound; that is, it needs enough of each of these resources but will hit its maximum throughput without exhausting either.
+
+This effect is much more pronounced on AWS than on GCP, due to the AWS TCP-to-VSOCK proxy, but exists on both platforms to some degree due to service architecture. 
+
+As a result, BFE tends to scale poorly vertically, but scales well horizontally. If you have exhausted the throughput of your BFE, we recommend you add another BFE of a similar size, and so on, to increase throughput and decrease latency. 
+
+On AWS, we recommend AdTechs start their dialing-in by reserving _half_ of the instance's vCPU cores and 4GB memory for running the parent, regardless of instance size. 
+
 ###### Bidding service
 
 The Bidding service usually requires higher compute power than the BFE to support
@@ -901,6 +927,12 @@ interest group.
   </tr>
 </table>
 
+Bidding Service Scaling Properties:
+
+For many workloads with which the B&A team has tested, Bidding Service is CPU-bound; that is, its throughput is scales primarily with available compute power. Bidding service thus scales well both vertically and horizontally. 
+
+On AWS, we recommend AdTechs start their dialing-in by reserving 2 vCPU cores and 4GB memory for running the parent, regardless of instance size. 
+
 ###### Determine BFE / Bidding instances ratio
 
 A **single** BFE service instance can support a higher RPS with multiple Bidding