Vlei Verifier Scaling

[aydarng]

To determine the optimal scaling strategy for the vlei-verifier, I recommend a structured approach divided into Foundation Steps and Solution Steps. The foundation steps focus on gathering critical information and understanding the application's limits, while the solution steps outline actionable strategies to address scaling challenges.

---

Foundation Steps

1. Define Non-Functional Requirements

Before implementing any scaling solution, it’s essential to establish clear non-functional requirements for the vlei-verifier in production. This includes:

• Expected Average RPS:
  What is the anticipated requests per second (RPS) the application needs to handle?

• Total Number of Users:
  How many users are expected to interact with the vlei-verifier in production?

• Data Retention Period:
  How long should presented credential data and account data be stored in the database? (Currently set to 20 minutes.)

These metrics will help estimate:

• RAM and CPU requirements for the environment.

• Storage size needed for the LMDB database.

Example:

• If the application handles 100 RPS and each credential record is 1 KB, the storage required for 20 minutes would be:

    100 RPS * 1 KB * 1200 seconds = 120 MB

---

2. Conduct Load Testing

To understand the vlei-verifier’s performance limits, conduct load testing:

• Objective:
  Determine the maximum RPS the application can handle without performance degradation.
  Measure request-response timeouts under varying loads.

• Metrics to Monitor:

  • CPU and memory usage.

  • Response times

  • Error rates under high load.

---

Solution Steps

Based on the findings from the foundation steps, I propose two distinct solutions to scale the vlei-verifier.

---

Solution 1: Single Pod Deployment with Vertical Scaling

This solution works within the constraints of LMDB by deploying a single instance of the vlei-verifier and scaling it vertically.

Steps:

1. Implement Single Pod Deployment

   • Deploy the vlei-verifier as a StatefulSet or Deployment with a replica count of 1.

   • Use a ReadWriteOnce (RWO) Persistent Volume (PV) for the LMDB database file

2. Configure Load Balancer

   • Create a Kubernetes Service of type LoadBalancer for the vlei-verifier application.

   • The AWS load balancer (ALB or NLB) will route traffic to the single vlei-verifier pod.

   • Configure health checks to ensure the pod remains healthy and responsive.

3. Optimize Vertical Scaling

   • Scale the application vertically by increasing the CPU and memory resources allocated to the pod.

   • Use larger instance types for the worker node hosting the pod to ensure sufficient resources.

---

Solution 2: Horizontal Scaling with vlei-router(or vlei-load-balancer)

This solution introduces an additional service, vlei-router, to enable horizontal scaling by routing traffic to multiple vlei-verifier instances, each with its own LMDB.

Steps:

1. Deploy Multiple vlei-verifier Instances

   • Deploy multiple instances of the vlei-verifier, each with its own dedicated LMDB database.

2. Introduce vlei-router

   • Deploy a new service, vlei-router, which acts as a load balancer and traffic router.

   • The orchestrator will route incoming requests to the appropriate vlei-verifier instance based on the requester’s AID (or SAID).

3. Configure Load Balancer for vlei-router

   • Create a Kubernetes Service of type LoadBalancer for the vlei-router.

   • The AWS load balancer (ALB or NLB) will route traffic to the vlei-router, which will then forward requests to the correct vlei-verifier instance.

---

Comparison of Solutions

Aspect	Solution 1: Single Pod Deployment	Solution 2: Horizontal Scaling with vlei-router
Scalability	Limited to vertical scaling.	Enables horizontal scaling.
Complexity	Simple to implement and manage.	More complex due to additional orchestrator logic.
Resource Utilization	Requires larger instance types for scaling.	Distributes load across multiple smaller instances.
Fault Tolerance	Single point of failure.	Improved fault tolerance with multiple instances.
Data Management	Single LMDB instance.	Multiple LMDB instances, each dedicated to a pod.

If after load tests we will see that the single vlei-verifier instance will meet Non-Functional Requirements we can use the 1st scaling approach since the vertical scaling will be enough in this case. Otherwise we will have to work on the 2nd scaling approach which will require the vlei-router implementation.

What do you think @2byrds, @ronakseth96 ?

[2byrds]

@aydarng thank you for introducing this discussion. I agree with a lot of your analysis. Some important wordings/PoV changes:

• Instead of defining non-functional requirements, I would say we have/will profile/analyze the current implementation for a given configuration (we might have a small set of variations of configurations that we analyze).
• That performance/configuration profile/analysis should be documented for a single instance, with the vertical scaling that you mention essentially being the modification to the configuration of the configuration of the single instance/deployment (and perhaps some different cpu/memory/etc settings). We have mostly done this already and only if there is some obvious gain to be had should be make changes and rerun the analysis of the single instance with multiple configurations.
• Kubernetes is a fine path for scaling. But we should not be mentioning a specific provider like AWS. So there won't be an AWS load balancer per se. We can generate some documentation related to the testing we do on AWS but in general our scaling/testing should be generic to the deployment environment since our customers will use a variety of providers.
• The goal of the horizontal scaling is to demonstrate that we can improve reliability and performance. So the analysis for the single instance is the starting point. The horizontal scaling should improve the reliability and/or performance and should inform our customers how to scale and what configuration is ideal, given what we have tested/documented.