[Milestone] Waku Network Can Support 10K Users

[jm-clius]

Priority Tracks: Secure Scalability
Due date: 31 May 2023
Milestone: https://github.com/waku-org/pm/milestone/5

Note: this deadline assumes that the target of 1 Mio users by end-June 2023 could lean for the largest part on the designed solutions for the problem space defined below.

Summary

• Scale to 10K Status Community users, spread across ~10 to ~100 communities
• This milestone focuses on 100% Desktop users, primarily using relay, but with experimental/beta support for filter and lightpush for clients with poor connectivity
• Communities, private group chats and 1:1 chats should be considered. Public chats are excluded.

Tasks / Epics

• 1. Verify scaling target requirements. Output: Scaling Status Communities : Potential Problems vacp2p/research#177 Status to implement
• 2. Community sharding plan Roadmap(SeM): Secure Scaling vacp2p/research#154 (note this issue also track 1mil work)
• 3. Simple Waku Relay DoS Mitigation Simple Waku Relay DoS mitigation vacp2p/research#164
• 4. Scalable storage: nwaku archive PostgreSQL implementation Waku Store Cache: PostgreSQL implementation #4 PostgreSQL logos-messaging-nim#1888
• 5. Scalable storage: Deterministic Message ID Calculating ID of the messages for status-go and deterministic WakuMessage bytes vacp2p/rfc#563 Waku message UID #9 (latter issue tracks work for distributed store, not part of 10k or 1mil milestones)
• 6. Scalable Storage: testing store at scale Roadmap(DST) vacp2p/research#191 (comment) Tracked as part of Configure fleet max connections status-im/infra-wakuv2#31
• 7. Filter and lightpush improvements Waku Filter and Lightpush: Evaluation and improvement #5
• 8. Peer management strategy Networking MVP: Refactor + extend functionality logos-messaging-nim#1353
• 9. Combine into comprehensive scaling strategy Roadmap(SeM): Application Protocols vacp2p/research#165
• 10. Targeted dogfooding, fleet setup and related docs docs: requirements for being a Waku fleet owner #61 Setting up static sharding fleet for Status logos-messaging-nim#1914
• 11. New multiaddrs discovery: libp2p rendezvous Discovery: discovering suitable multiaddrs vacp2p/research#176
• 12. Waku static sharding implementation [Epic] Waku Relay Topic Sharding: Static Sharding Implementation #15 (issue also tracks js-waku work with is not in scope for 10k milestone)
• Service/Capability Discovery
• 1k nodes simulation https://discord.com/channels/1110799176264056863/1135615797700608090/1135615800510795888
• Automate release and build distributable binaries logos-messaging-nim#611

---

Extracted questions

• Are the number of users and number of communities realistic? Answer on 2023-01-19: yes, makes sense as an initial goal
• What is the proportion (in message rate and bandwidth) of community messages vs community control messages vs store query-responses?
• Does message rate increase linearly with increase in network size? Answer on 2023-01-19: generally should be the case (could have a multiplicative factor, but not combinatorial or exponential)_
• What bandwidth upper bound should we target for Desktop nodes? One possible answer: ADSL2+ limit of 3.5 Mb/s?
• Can this MVP consider participation in only one Community at a time? Answer on 2023-01-24: nodes will be part of multiple communities from the beginning.
• What store query rate should we target for 10K users?

Network requirements

Note: this gathers the minimal set of requirements the Waku network must adhere to to support Status Communities scaling to 10K users. It does not propose a design.

1. Message Delivery and Sharding

Note: this section, especially, depends on app-defined user experience minimals. E.g. the app knows what (sub)set of messages is necessary "for a consistent experience" and this will feed into a pubsub topic, content topic and sharding design that does not compromise on UX. This process should also define when messages should be received "live" (relay) or opportunistically via history queries (store).

1. Nodes should be able to receive (via relay or store) all community messages of the community they're part of.
2. Nodes should receive live (via relay) all chat messages that is necessary for a consistent experience. A chat message is content sent by users either in a community channel, 1:1 or private group.
3. Nodes should receive live (via relay) all control messages that is necessary for a consistent experience. Control messages are mostly used for community reasons, with some for 1:1 and private groups (e.g. online presence and X3DH bundle).
4. Each community can utilize a single or multiple shards for control and community messages, as long as requirements (1) - (3) still hold.
5. Nodes should participate in shards in such a way that resource usage (especially bandwidth) is minimized, while requirements (1) - (3) still hold.
6. Peer and connection management should be sufficient to allow nodes to maintain a healthy set of connections within each shard they participate in.

Assumptions:

• connectivity, NAT traversal capability, NAT hole punching, etc. is similar to that described Status MVP: Status Core Contributors use Status #7. No further work is required within the context of this MVP.
• it is possible to be part of several communities simultaneously
• we assume that community size is such that community desktop nodes can realistically be expected to relay the messages for all community traffic. That is - communities can be responsible for their own relay infrastructure.

2. Discovery

1. Nodes should be able to discover peers within each shard they're interested in.
2. Discovery method(s) can operate within a single or multiple shards, as long as:

• requirement (1) still holds
• nodes can bootstrap the chosen discovery method(s) for shards they're interested in
• the chosen discovery method(s) does not add an unreasonable resource burden on nodes, especially if this method is shared between shards

Assumptions:

• nodes are able to use discv5 as their main discovery method

3. Bootstrapping

1. Nodes should be able to initiate connection to bootstrap nodes within the shards they're interested in.
2. Bootstrap nodes can serve a single or multiple shards, as long as they can handle the added resource burden.

Assumptions:

• Status initially provides bootstrapping infrastructure.
• DNS discovery is sufficient to find initial bootstrap nodes.

4. Store nodes (Waku Archive)

1. Nodes should be able to find capable store nodes and query history within the shards they're interested in.
2. Store nodes can serve a single or multiple shards, as long as:

• they can handle the query rate and resource burden
• are discoverable as stated in requirement (1)

Assumptions:

• Status provide initial store infrastructure, including a performant Waku Archive implementation.
• PostgreSQL implementations exist for Waku Archive that can deal with the required rate of parallel queries to support 10K users
• DNS discovery is sufficient to discover capable store nodes (these may or may not be the same nodes as used for bootstrapping, but discovery will be simpler if they are).

5. Security:

1. Community members should not be vulnerable to simple DoS/spam attacks as defined in (3) and (4) below.
2. Each community should be unaffected by failures and DoS/spam attacks in other communities. This implies some isolation/sharding in the messaging domain.
3. Store/Archive:
   • store nodes for a community should only archive messages actually originating from the community
   • store nodes for a community should not be vulnerable to being taken down by a high rate of history queries
4. Relay:
   • community relay nodes should only relay messages actually originating from the community.

Assumptions:

• Community members (i.e. the application) are able to validate messages against community membership.

Other requirements

Note: this gathers the minimal set of requirements outside the Waku network (e.g. operational, testing, etc.) to support Status Communities scaling to 10K users.

1. Kurtosis network testing

A simulation framework and initial set of tests that can approximate:

• the protocols
• the discovery methods
• the traffic rates for a typical community
  in such a way to prove the viability of any scaling design proposed to achieve the Network Requirements

2. Community Protocol hardening

The Community Chat Protocols specifications are moved to Vac RFC repo.

• what else is required within this MVP time frame, e.g. including Community Chat in Kurtosis testing?

3. Nwaku integration testing

Nwaku requires integration testing and automated regression testing for releases to improve trust in stability of each release.

4. Fleet ownership

Ownership for infrastructure provided to Status communities should be established. This may require training and transfer of responsibilities which mostly lies de facto within the nwaku team.
Fleet ownership comprises the responsibility for:

• establishing a sensible upgrade process (may require some nodes for staging)
• upgrading fleets
• monitoring existing fleets and protocol behavior
• support and logging bugs when noticed

Initial work

The requirements above will lead to a design and task breakdown. Roughly the order of work:

Ownership for all three items below is shared between Vac, Waku and Status teams:

(1) Agree on requirements above as the complete and minimal set to achieve the 10K scaling goal.
(2) A viable, KISS network design adhering to "Network requirements"
(3) Task breakdown of each item and ownership assignment

[corpetty]

tagging Testing team: @AlbertoSoutullo, @0xFugue, @Daimakaimura

[jm-clius]

Achieving network requirements: tasks and ownership

NB: requirements and tasks may change as we encounter unknowns. The task breakdown below assumes that nothing more has to be done for Discovery and Bootstrapping other than proper configuration (to be described in the Scaling Strategy BCP).

1. Verify scaling target requirements

Understand the expected:

• rate of chat messages
• rate of control messages
• rate of store queries
• avg expected bandwidth usage of each of the above

for 10K community users.

Note: this is not necessarily an analytical exercise but ballpark figures and sanity checking current Status Community message rates. @Menduist has done analysis of message rates in large Discord servers to get to rough estimate of what we would expect to see for Status Communities. However, analysis of existing Status Community shows significantly higher message rate and bandwidth usage. See conversation.

Tracked in: ??
Owners:

• app: @richard-ramos
• waku: @rymnc

2. Community sharding plan

Sharding strategy for Waku relay in general and Status Communities specifically. This plan will consider short term and longer term strategies.
This item is set out in more detail in @kaiserd's Secure Scaling Roadmap.

Tracked in: vacp2p/research#154
Owners:

• app (strategy and implementation): @cammellos
• waku (strategy): @kaiserd

3. Simple Waku Relay DoS mitigation

Strategy and implementation to protect relay and store against simple DoS attack vectors. This item is set out in more detail in @kaiserd's Secure Scaling Roadmap.

Tracked in: vacp2p/research#164
Owners:

• app (implementation): @cammellos
• waku (strategy): @alrevuelta
• nwaku (implementation): @alrevuelta

4. Scalable storage: nwaku archive PostgreSQL implementation

Already part of #8 but repeated here for completeness.
Note that this includes work to allow concurrent queries.

Tracked in: #4
Owners:

• postgresql backend: @cammellos
• postgresql chronos/concurrency adaptation: ??

5. Scalable storage: deterministic message ID

Tracked in: vacp2p/rfc#563
Owners:

• waku (protocol): @LNSD
• nwaku (implementation): @LNSD
• go-waku (implementation): @richard-ramos

6. Scalable storage: testing store at scale

Basic testing to see that PostgreSQL implementation works at expected message and query rates. (Note this is in addition to simulation with Kurtosis).

Tracked in: ??
Owner: @LNSD

7. Filter and lightpush improvements

Revising the RFCs and implementations in nwaku and go-waku. Already part of #8 but repeated here for completeness.

Tracked in: #5
Owners:

• protocol: @jm-clius
• nwaku (implementation): @jm-clius
• go-waku (implementation): @richard-ramos

8. Peer management strategy

RFC for basic peer management strategy and implementations in nwaku and go-waku.

Tracked in: logos-messaging/logos-messaging-nim#1353
Owners:

• protocol (RFC): @alrevuelta
• nwaku (implementation): @alrevuelta
• go-waku (implementation): @richard-ramos

9. Combine into comprehensive scaling strategy

This can be seen as the final goal for all the moving parts and separate tasks listed above. Output will likely take the form of one or more Best Current Practices RFCs that focus on the Status 10K use case. It will bring together the short term strategies for sharding, DoS mitigation, bootstrapping, discovery and store configuration. It may include suggestions on when to use lightpush and filter rather than relay.

Tracked in: vacp2p/research#165
Owners:

• protocol (RFC): @kaiserd
• app (strategy consultation): @cammellos

10. Targeted dogfooding

This is in addition to simulation with Kurtosis. Individual owners of each task will be responsible for testing and dogfooding their strategies/features. This task ensures that we have considered each item for targeted network testing, including:

• Sharding strategy
• Store at scale
• Bootstrapping
• Discovery (DNS, discv5 and Waku Peer Exchange)
• Filter and lightpush

Tracked in: ??
Owner: @jm-clius

11. New multiaddrs discovery: libp2p rendezvous

Although it is possible to encode multiaddrs in ENRs, which are currently being exchanged by all existing discovery methods, ENRs are limited in size and can consequently not contain more than one or two multiaddrs. We need a discovery method more suitable for multiaddrs. We have chosen libp2p rendezvous as solution here.

Tracked in: vacp2p/research#176
Owners:

• nwaku: ??
• go-waku: @richard-ramos

12. Waku static sharding implementation

This is an outflow of the Community sharding plan as specified by @kaiserd and covers the implementation portion, including configuration and enabling shard discovery via ENRs.

Owners:

• nwaku: ??
• go-waku: @richard-ramos

[jm-clius]

Achieving other requirements: tasks and breakdown

1. Wakurtosis: first network test

This is described in vacp2p/wakurtosis#7

It covers testing the scalability of the relay protocol, specifically measuring:

• latency
• reliability
• resource usage (bandwidth, CPU, memory)
  against network size and message rate.

Owner:

• data science: @Daimakaimura

2. Wakurtosis: analyze first test results

This step will either confirm our (positive) assumptions about relay scalability or highlight bottlenecks/bugs in the protocol or implementations, which must be addressed and considered in the overall network roadmap.

Owners:

• waku: @fryorcraken
• data science: @Daimakaimura

4. Wakurtosis: plan next tests

This is a collaborative task flowing from the results of the first test to refine the simulation(s) and plan the next, most useful tests.

Owners:

• waku: @jm-clius
• data science: @Daimakaimura

5. Community Protocol: move to Vac RFC repo

This is an administrative step. It may require updating the RFC to match the latest implementation, moving sections around, etc.

Owner:

• waku: @rymnc

6. Community Protocol: review protocols

Grasping the content of each protocol and how it maps to real-world Waku network traffic. This is potentially an involved task, so the scope should be minimized for this MVP. This relates to Verify scaling target requirements under the Network Requirements.

Owner:

• waku: @rymnc
• app: @cammellos (inputs from @richard-ramos @felicio)

7. Nwaku hardening: Wakurtosis sandbox machine

Provisioning a performant machine(s) which the dev team can use for sandbox testing features using ad-hoc Wakurtosis deployments.

Owners:

• waku (requirements): @jm-clius
• infra (devops): ??

8. Nwaku hardening: Wakurtosis integration testing

Integration test environment for nwaku. Most likely it will take the form of a pipeline that deploys a Wakurtosis network topology and runs a series of scripted integration tests for nwaku.

Owners:

• waku (requirements): @jm-clius
• infra (devops): ??

9. Nwaku hardening: release automation

Automated release pipeline for nwaku that builds a release, compile release notes and publish release binaries and tagged docker image for most common OSs/architectures.

Tracked in: logos-messaging/logos-messaging-nim#611
Owners:

• waku (requirements): @jm-clius
• infra (devops): ??

10. Fleet ownership: set requirements

Create a document that summarizes all the common tasks that a fleet owner generally has to do, including deployment, monitoring and debugging. This will also allow us to communicate to other platforms planning on deploying their own Waku fleets what they need to consider. The document should include a section on what Status fleet ownership specifically entails, including a procedure to log and escalate bugs/network anomalies.

Owner: @jm-clius

11. Fleet ownership: training

Based on the requirements determined above, determine who will take ownership of the Status fleets and schedule training sessions.

Owner:

• waku: @jm-clius
• Status fleet owners: ??