chore: review waku-simulator deployment and improve tracking processes

[gabrielmer]

Background

We have a waku-simulator deployment running in https://simulator.waku.org/
We've noticed that the amount of connected peers has dropped lately

Details

We need to confirm that the simulator is constantly getting updated to latest master version, investigate the root cause of the peers disconnections, and implement processes and/or Grafana alarms to promptly become aware whenever there's unexpected behaviors in the simulation.

Acceptance criteria

• check that latest master's version is getting automatically deployed
• return simulation to expected behavior
• devise strategy to keep track of simulation's status

[gabrielmer]

Noticed that nwaku's latest image isn't getting updated in DockerHub with every merge and therefore, waku-simulator's version isn't getting synced with latest master.

Requested infra team for latest image to update with every merge. Will drop a comment once it's working and waku-simulator is properly synced with master

[chair28980]

@gabrielmer given the background description do you feel like this qualifies as a bug? -- If not perhaps we can find an appropriate 2024 Milestone to group this work into.

[gabrielmer]

@gabrielmer given the background description do you feel like this qualifies as a bug? -- If not perhaps we can find an appropriate 2024 Milestone to group this work into.

mmm I don't think it qualifies as a bug. Most of the work is related to improving or even creating a process, and not about something broken in an existing process. The following milestone might be appropriate https://github.com/waku-org/pm/milestone/3

[gabrielmer]

Implemented alerts for the simulator in logos-messaging/logos-messaging-simulator#46 but not merging.

Was made aware that the simulator is not in a phase right now to be regularly monitored. Once it's stable, the respective processes to track the simulator will be put in place.