[Task] Browser simulations

[weboko]

Description

Provide tools for DST team to perform simulations using js-waku.

Requirement (from FURPS)

In an established network with 1 bootstrap node, 100 service nodes and 500 browser nodes (all nodes have a limit of 100 connections and support WebSocket), a new browser node can start sending and receiving messages within 10 seconds

User Story

• As DST engineer, I am capable to integrate js-waku into simulations flow, so that I can make it connect to set nwaku node OR fleet.
• As DST engineer, I can extract useful data from js-waku node after concluding simulation, so that I can analyze it.
• As DST engineer, I can capture connectivity issues, so that they can be measured and reported.
• As DST engineer, I can identify missed messages on Filter, so that they can be measured and reported.
• As DST engineer, I can identify failed LightPush requests, so that they can be measured and reported.
• As DST engineer, I can identify missed messages on Store, so that they can be measured and reported.
• As DST engineer, I can capture libp2p information, so that it can be integrated in overall report.
• As Waku engineer, I can get access to data and derive useful information, so that I can drive improvements.

Proposed Solution

Provide dockerized web based application capable to be run in headless browser.
Example that can be used: https://github.com/waku-org/js-waku/tree/master/packages/browser-tests

In dockerize application we must expose way for measuring successes / failures in engineering AND logical sense in respect to the item:

• Connectivity;
• Stream / Peer management;
• Store protocol;
• Filter protocol;
• LightPush protocol;
• Discovery (?);

We MUST ensure that dockerized application is configurable by environment for particular fleet / nwaku node or any other needed configuration for simulations.

We MAY need to integrate following packages to provide easy way for libp2p metrics exposure:

• https://github.com/libp2p/js-libp2p/tree/main/packages/metrics-devtools
• https://github.com/libp2p/js-libp2p/tree/main/packages/metrics-opentelemetry
• https://github.com/libp2p/js-libp2p/tree/main/packages/metrics-prometheus
• https://github.com/libp2p/js-libp2p/tree/main/packages/metrics-simple

We also MAY need to provide a way to capture js-waku specific information such as discovery(if it is part of simulations), peer management, etc.

[AlbertoSoutullo]

In order to do the User Story points, as a DST engineer, I would need to:

• Be able to configure to adjust node configuration options either by command line or env variables
• Be able to access the /metrics endpoint (for healthiness, connections, ...)
• Be able to access the REST API endpoint (for publishing messages, store queries, filter and lightpush operations, ...)
• Be able to extract (if it is by logging, better for us) when a message is sent or received, either by relay, filter or lightpush, with the following information: msg_hash, timestamp, receiver_peer_id, sender_peer_id. If it can have the same format as nwaku, better for us as well.

[adklempner]

https://discord.com/channels/1110799176264056863/1348999957847216210/1358740047456636999

So, to check logs we use regex patterns like the following ones:

r'received relay message.*?my_peer_id=([\w*]+).*?msg_hash=(0x[\da-f]+).*?from_peer_id=([\w*]+).*?receivedTime=(\d+)'

r'handling lightpush request.*?my_peer_id=([\w*]+).*?peer_id=([\w*]+).*?msg_hash=(0x[\da-f]+).*?receivedTime=(\d+)'

r'sent relay message.*?my_peer_id=([\w*]+).*?msg_hash=(0x[\da-f]+).*?to_peer_id=([\w*]+).*?sentTime=(\d+)'

Of course this can be adapted, but it would be incredibly helpful that all waku implementations follow the same logging format.

Regarding endpoints, the ones we mostly use are:

• Get v1 info for getting ENRs
• Post v1 lightpush message for injecting messages via nwaku lightpush
• Get v3 storage for getting messages from store
• Get v2 filter messages for getting messages from filter.