Prometheus' self metrics should be scraped and remote written into Prom by Otelcol

[sinapah]

Enhancement Proposal

We define quite extensive bundled alert rules for Prometheus. In COS Lite, we rely on Prometheus' self monitoring to monitor the health of Prom itself. However, one disadvantage of this approach is that we don't get the benefit of those bundled rules and Prom only relies on the generic alert rules coming from cosl. To get the benefit of these bundled rules, we should have Otelcol scrape Prom and remote write the scraped metrics into Prom, similar to what we do with Mimir in COS.

The way I see it, there are scenarios for how Prometheus can be deployed. The list below outlines what we should do in each:

1. Prometheus on its own in non-standard COS deployments: if we just have Prom on its own, we can continue doing what we do already - that is, use Prom's self-mon to monitor health.
2. If we have Prometheus inside COS Lite AND there is a self-metrics-endpoint relation, the Prometheus charm should NOT write a self-mon scrape job and instead, self monitoring will come from Otelcol.

[MichaelThamm]

So, you suggest something like this?

if not any(self.model.relations.get("self-metrics-endpoint", [])):
	self._configure_self_mon()

I think this makes sense to avoid the duplicate metrics we would see if we allowed both