Better operationalize the FilOz-run f3-observer and f3-aider

[BigLep]

Done Criteria

There is good monitoring/alarming on observer/adder that FilOz runs so we are notified proactively if there are any production issues.

Why Important

filecoin-project/lotus#13208 revealed that the FilOz-run observer instance had silently not been collecting data for a month due to storage space being filled up. This in turn mean the f3-aider instance wasn't helping with rebroadcasting messages. This in turn helped lead to F3 not making progress for multiple days.

User/Customer

Implementers who will get called in if F3 stops making progress.
Overall network participants who care about faster finality.l

Tasks

• Add monitoring for the f3-observer storage space. We should get a slack notification if storage fills up to 80%.
  • Effectively addressed by the monitors described in Better operationalize the FilOz-run f3-observer and f3-aider #1041 (comment)
• Add monitoring for the f3-observer process. If the process isn't emitting a healthy heartbeat, then we should alarm.
  • Effectively addressed by the monitors described in Better operationalize the FilOz-run f3-observer and f3-aider #1041 (comment)
• Add monitoring for the f3-aider process. If the process isn't emitting a healthy heartbeat, then we should alarm.
  • We decided not to do this per Better operationalize the FilOz-run f3-observer and f3-aider #1041 (comment)
• (bonus) Setup storage pruning if the f3-observer storage is filling up.
  • Handled in feat: Add retention size limit for rotated files #1040 and https://github.com/filecoin-project/filoz-infra/pull/346

Notes

1. The actual infrastructure for this will run in https://github.com/filecoin-project/filoz-infra, but this issue was created in go-f3 since that repo is public and to give better visibility to the work.

[BigLep]

We're going to mark this as resolved.

f3-observer retention limits

This was added in #1040 and deployed to FilOz infrastructure in https://github.com/filecoin-project/filoz-infra/pull/346

f3-observer monitoring

We have metrics/alarms on the f3-observer process that FilOz runs to make sure we're storing data on disk and that we're receiving messages. If the process dies or runs out of disk space, these alarms would both trigger. If the process is running but not connecting to peers, the second alarm would trigger. Alarms post to a FilOz slack channel.

https://grafana.f3.eng.filoz.org/alerting/grafana/best5rk6ieozkb/view

https://grafana.f3.eng.filoz.org/alerting/grafana/cest7nej1ts00c/view?returnTo=%2Falerting%2Flist&tab=query

f3-aider

we've decided not to invest in monitoring this process because:

1. It's stateless. f3-observer is where the state is.
2. It's a very simple process. It's hard to imagine a situation where the process is running and not doing its job (assuming the observer has messages to broadcast)
3. Kubernetes will restart it if it dies, and it's hard to imagine a scenario where it would get in a die/reboot loop (i.e., there being some message from the f3-observer that kills the process).
4. This is temporary. After the nv27 network upgrade, we expect to be able to shut down this software since node software will have more resiliency.

Other

The issue description was updated.
This filoz-infra was documented in https://www.notion.so/filecoindev/F3-Operational-Excellence-5cdce4f1aa6e4c398b475f6e690c47fe?source=copy_link#23adc41950c180efbdcac9057df36b18