Scaling the Waku Protocol: A Performance Analysis with Wakurtosis

rlog/2023-08-04-wakurtosis-waku-scallability.mdx

@@ -13,42 +13,57 @@ categories: waku, wakurtosis
 
 <!--truncate-->
 
-### Introduction
+## Introduction
+The scalability and performance of the Waku protocol are subjects of critical importance to our team. To explore these facets in depth --- i.e. high granularity --- we turned to Wakurtosis, a bespoke simulation framework developed by the VAC-DST team. By studying various network sizes, message rates, and peer discovery setups we aimed to better understand the protocol's capabilities and limitations, hence aspects that could benefit from further.
 
-The scalability and performance of the Waku protocol are subjects of critical importance to our team. To explore these facets in depth --- i.e. high granularity, we turned to Wakurtosis, a bespoke simulation framework developed by the VAC-DST team. By studying various network sizes and message rates, we aimed to gain valuable insights into the protocol's capabilities and areas that could benefit from further refinement.
+## Understanding Wakurtosis
+Wakurtosis is a robust simulation framework which integrates Docker and Kurtosis to create a simulation environment which allows to run highly granular, large scale simulations with a variety of traffic and network patterns. At the core of Wakurtosis is Kurtosis — an orchestration tool responsible for managing containers, known as services, within isolated environments called enclaves. These enclaves house virtual networks and their respective containers. In addition to this, several external modules developed in-house expand some of Kurtosis's limited functionalities:
+- Network Generation Module (Gennet): Initiates and configures networks for the simulation. It's highly modular, supporting the integration of multiple protocols and node traits.
+- Packet Injection Module (WLS): Allows for the insertion of custom data packets, thereby enabling varied traffic patterns and stress tests on the simulations.
+- Analysis Module: Captures and provides insights into resource usage, network behaviors and protocol interactions throught the enclave.
 
-### Understanding Wakurtosis
+### Data Collection
+Wakurtosis ensures the accuracy of its data by leveraging multiple sources for hardware metrics:
 
-Wakurtosis is a key tool in our analysis, designed specifically for large-scale testing and simulation of blockchain protocols like Waku. It consists of three essential modules:
+##### Cadvisor (a Google tool)
+Cadvisor providess detailed metrics on resource usage and performance characteristics of Docker containers. Cadvisor monitors application containers at the individual level by directly interfacing with Docker's daemon API. While Cadvisor offers real-time metrics, it primarily focuses on container-specific metrics, which may neglect broader system-level insights.
 
-- Network Generation Module (Gennet): This takes care of generating and initializing networks for simulation.
-- Packet Injection Module (WLS): Responsible for injecting custom data packets to simulate different traffic patterns.
-- Analysis Module: Processes the simulation data to provide an in-depth understanding of network behaviors.
+##### Docker statistics
+Docker statistics provides insights into Docker's overall performance and resource allocation. This native Docker tool captures statistics about running containers using Docker's stats API, collecting cumulative CPU, memory, network, and block I/O metrics. Docker statistics offer a bird's-eye view of the system, which can sometimes miss the granularity of performance fluctuations inside individual containers, particularly when dealing with multiple processes per container.
 
-Data sources like Cadvisor/Prometheus, Docker Stats, and process-level monitoring enhance the reliability of Wakurtosis, offering a comprehensive view of system performance.
-
-### Examining the Waku Protocol
+##### Process-level monitoring
+Process-level monitoring offers detailed insights by tracking operational traits of processes within containers. This method employs deep inspection of individual processes running inside a container by accessing */proc* kernel files to gather real-time process statistics. Reading from the */proc* filesystem offers a direct window into kernel statistics, providing comprehensive metrics on each process within the containers. However, while it offers granular insights, process-level monitoring can be more resource-intensive and might not always capture overarching system behavior.
 
-Despite certain scalability challenges, Wakurtosis has proved effective in simulating Waku with over 1,000 nodes on a single machine. Our analysis under normal operating conditions revealed promising scalability from a memory standpoint. However, there was evidence of increased memory usage with higher message rates, indicating potential areas for optimization.
+### Performance Metrics
+Data sources like Cadvisor/Prometheus, Docker Stats, and process-level monitoring enhance the reliability of Wakurtosis, offering a comprehensive view of system performance.
 
-Bandwidth utilization exhibited minor fluctuations as the network size increased, and efficiency in bandwidth usage improved with an increase in the message rate. Some inefficiencies were noted in larger networks, suggesting room for improvement.
+- Hardware Level Metrics: Emphasis on memory usage, CPU consumption, Disk I/O, and Network I/O.
+- Topology Level Metrics: Focuses on the efficiency of message propagation across the network, including metrics like message injection, propagation times, and message loss. 
 
-### The Impact of Discovery v5 on Waku
+### Scalability
+To overcome scalability challenges, Wakurtosis employs a multi-node approach, running several nodes within one container. This method supports simulations with over 1,000 nodes on a single machine. However, this can introduce unforeseen network effects, potentially affecting some metrics. Nevertheless, Wakurtosis offers the flexibility to choose between this and a 1-to-1 simulation, catering to the specific needs of each test scenario.
 
-The introduction of Discovery v5 to the Waku protocol led to noticeable changes in performance. Memory consumption and bandwidth usage rose with the network size and message rate, pointing to additional overhead due to Discv5. The decreased efficiency in data transmission and reception suggests that Discv5’s overhead significantly affects the protocol's performance.
+## Examining the Waku Protocol
 
-### Future Directions
+### Simulation Setup
+To evaluate Waku under varied conditions, we conducted simulations across a range of network sizes, topologies, and message rates. Each simulation lasted 3 hours to allow the network to stabilize. The network sizes explored included 75, 150, 300, and 600 nodes. For non-Discv5 simulations, we used static topologies with average node degrees of *K=3*, *K=13*, and *K=50*. For Discv5 simulations, we configured max_peers settings of 12 and 50 to approximate similar average degrees. To stress test message throughput, we simulated message rates of 1, 10, and 100 messages per second. This allowed us to observe performance under low, moderate, and high traffic loads. We leveraged Wakurtosis' ability to run multi-node simulations to achieve networks at the scale of 600 nodes. This combination of network sizes, topologies, message rates, and hardware configurations enabled us to comprehensively evaluate Waku's performance and scalability boundaries under diverse conditions.
 
-Our analysis has helped identify limitations in the existing infrastructure supporting the Waku protocol. To handle high-scale simulations and high-rate traffic more effectively, enhancements in infrastructure are required.
+### Results
+Despite certain scalability challenges, Wakurtosis has proved effective in simulating Waku with over 1,000 nodes on a single machine. We examined the behaviours of the Waku protocol with multiple network sizes, message rates and peer discovery strategies. Our analysis under baseline conditions --- static network with average degrees of *K=3* and *K=13* --- revealed promising scalability and stability from a memory standpoint. However, there was evidence of increased memory usage with higher message rates, indicating potential areas for optimization. Bandwidth utilization exhibited minor fluctuations as the network size increased, and efficiency in bandwidth usage improved with an increase in the message rate. Some inefficiencies were noted in larger networks, suggesting room for improvement but overall Waku displayed very stable behaviour in terms of memoery and bandwidth usage.
 
-The absence of message loss in our simulations bodes well for the protocol’s robustness. These findings, coupled with the insights gained from the study, will guide our continuing development and optimization efforts for the Waku protocol, particularly in high-traffic scenarios.
-Conclusion
+![Bandidth effiencies multipliers over pure payloads in transmission](/static/img/wakurtosis_waku/waku_bandwidth_rx_efficiency.png)
+![Bandwidth efficiencies multipliers over pure payloads in reception](/static/img/wakurtosis_waku/waku_bandwidth_tx_efficiency.png)
 
-The scalability and performance of the Waku protocol are central to its ongoing development. Our study, utilizing Wakurtosis, has provided a nuanced understanding of these aspects. As we look ahead, we are focused on creating a protocol that is efficient, resilient, and scalable to meet diverse application needs. Keep an eye on our blog for more updates on our progress.
+#### Impact of Discovery v5
+<!--When incorporating Discovery v5 into the Waku protocol, we initially observed marked differences in performance metrics, particularly concerning memory consumption and bandwidth usage. However, after adjusting our non-Discv5 simulation parameters to better mirror the natural conditions of Discv5 (namely, by focusing on an average degree closer to 15), these disparities became less pronounced.-->
 
-### References
+<!--Although there's still evidence of Discv5 introducing overhead, the updated simulations indicate that the gap in performance between the two configurations is much narrower than initially believed. It's worth noting that while Discv5's design naturally results in increased resource utilization, especially with growing network size and message rate, it may not necessarily translate to inefficiencies in the protocol. Instead, it could be seen as an inherent trade-off for the benefits offered by Discovery v5.-->
 
+## Conclusion
+This study underscores the Waku protocol's resilience and scalability across varied conditions, but also highlights the need for a more robust simulation infrastructure for demanding Wakurtosis scenarios. The protocol's robustness, evidenced by the absence of message loss, notable stability across network sizes and traffic loads, is a notable takeaway. Guided by these insights, our immediate priority is to keeping studying Waku behaviour for greater scalability and performance, especially under larger network loads, high-traffic situations, and different protocol configurations. Amidst the dynamic landscape of decentralization, our dedication to refining Waku remains unwavering. Stay updated with our progress.
+## References
 - [Tech Report](https://docs.google.com/document/d/1U3bzlbk_Z3ZxN9tPAnORfYdPRWyskMuShXbdxCj4xOM/edit?usp=sharing)
+- [Tech Report Addendum](https://docs.google.com/document/d/18gU7Pxn7eBBwhtlj7kz4zbRR_Ae7txaUexo3MJQLbIY/edit?usp=sharing)
 - [Wakurtosis](https://github.com/vacp2p/wakurtosis)
 - [Waku](https://waku.org/)
 - [Discovery v5](https://rfc.vac.dev/spec/33/)