Consolidate process count metrics

[jon-signal]

What does this PR do?

This introduces a new, single foundationdb.processes_per_role gauge that is tagged by fdb_role and fdb_process_class. It makes no changes to existing metrics.

Motivation

There are two main motivations for this new gauge:

1. The set of role types in a FoundationDB cluster can change from version to version. For example, FoundationDB 6.x had a notion of a proxy role, but in FoundationDB 7.x, that role is gone and has been replaced by grv_proxy and commit_proxy. Using tags (instead of individual gauges) to count processes by role gives this integration more flexibility to work with different versions of FoundationDB.
2. Having processes tagged by role gives operators a mechanism by which they can detect if a single process is performing multiple roles. For example, in a heavily-loaded cluster, having a single process act in both the storage and log roles would be bad news, and operators might want to add a monitor for problematic duplicate role assignments.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add the qa/skip-qa label if the PR doesn't need to be tested during QA.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[jon-signal]

Note that foundationdb.instances was previously a count, not a gauge, but I think that was a bug.

[jon-signal]

Following up, I still think this is a bug, but out of scope for this pull request.

[codecov]

Codecov Report

Attention: Patch coverage is 86.95652% with 3 lines in your changes missing coverage. Please review.

Project coverage is 85.36%. Comparing base (3cabf9e) to head (2959875).
Report is 3 commits behind head on master.

Additional details and impacted files

Flag	Coverage Δ
activemq	?
cassandra	?
foundationdb	82.77% <86.95%> (+0.24%)	⬆️
hive	?
hivemq	?
hudi	?
ignite	?
jboss_wildfly	?
kafka	?
presto	?
solr	?

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jon-signal]

Thinking about this a little more, this is definitely going to need dashboard updates, which I simply haven't done yet. I guess let's see if we agree on the basic idea first; if the general approach seems reasonable, then I'll dive into the dashboards/alerts/everything else!

Also, I think we can actually collapse this down into a single foundationdb.processes gauge tagged by fdb_process_class fdb_role. This depends on my understanding of repeated tags (for processes with multiple roles), though, which is admittedly a little shaky. The idea here is that we might wind up with something that looks like this:

foundationdb.processes = 21 (['fdb_process_class:stateless'])
foundationdb.processes = 4  (['fdb_process_class:log', 'fdb_role:coordinator', 'fdb_role:log'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:data_distributor'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:cluster_controller'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:consistency_scan'])
foundationdb.processes = 9  (['fdb_process_class:storage', 'fdb_role:storage'])
foundationdb.processes = 5  (['fdb_process_class:log', 'fdb_role:coordinator'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:grv_proxy'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:ratekeeper'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:master'])
foundationdb.processes = 2  (['fdb_process_class:stateless', 'fdb_role:commit_proxy'])
foundationdb.processes = 1  (['fdb_process_class:stateless', 'fdb_role:resolver'])

I think what will happen is that just querying foundationdb.processes with no tags will yield the number of fdbserver processes with no double-counting. At the same time, if we ask for the sum of all foundationdb.processes values tagged with fdb_role:coordinator, we should wind up with 9 (the 5 stand-alone coordinators and then the 5 in the dual role of log and coordinator). I think that means that we can use that single metric to do the job that was previously done by two metrics and then an additional family of metrics.

The other nice thing about the "tag by role" approach (and I should have mentioned this earlier) is that it means that if new roles appear in future versions of FoundationDB, those new roles will just appear as new tags and operators won't have to add new metrics to their "how many processes do I have?" queries.

Seem reasonable?

[jon-signal]

From the conversation in #19681 (comment), it sounds like the best way forward here would be to introduce a new metric rather than changing anything that already exists. I'm not in a position to do that this afternoon, but acknowledge it needs to get done and will address it as soon as I can!

[jon-signal]

I've revised this significantly in light of discussions elsewhere, and now I think it's much less controversial. I'll update the description to match.

[steveny91]

@jon-signal Hello again! I'll take a look at this soon ™️