Expose All Metrics in tokio::runtime::RuntimeMetrics?

[AlJohri]

I've made the mistake a few times now of thinking a metric doesn't exist and finding it is in tokio::runtime::RuntimeMetrics rather than this crate.

Some examples of this are:

• active_tasks_count
• num_blocking_threads
• num_idle_blocking_threads
• blocking_queue_depth
• io_driver_fd_registered_count
• io_driver_fd_deregistered_count

Is there a way to expose all of the available tokio metrics here and keep them in sync?

Currently we use tokio-metrics for most metrics and then manually get the missing ones from the runtime handle:

struct AdditionalTokioRuntimeMetrics {
    active_tasks_count: usize,
    num_blocking_threads: usize,
    num_idle_blocking_threads: usize,
    blocking_queue_depth: usize,
}

// NOTE: some metrics are not available from `tokio_metrics::RuntimeMetrics` so instead we
// get it from `tokio::runtime::RuntimeMetrics`
fn get_additional_tokio_metrics(
    runtime: tokio::runtime::RuntimeMetrics,
) -> AdditionalTokioRuntimeMetrics {
    AdditionalTokioRuntimeMetrics {
        active_tasks_count: runtime.active_tasks_count(),
        num_blocking_threads: runtime.num_blocking_threads(),
        num_idle_blocking_threads: runtime.num_idle_blocking_threads(),
        blocking_queue_depth: runtime.blocking_queue_depth(),
    }
}

Hoping to remove this snippet of code and rely exclusively on tokio-metrics.

[AlJohri]

Related: https://github.com/tokio-rs/tokio-metrics/pull/49/files

[mox692]

I don't see any harm in adding these metrics as well.

Note that some of these metrics are not stabilized yet, so the they could be changed or removed. (See also tokio-rs/tokio#6546.)

[arielb1]

These 4 sound good:
active_tasks_count
num_blocking_threads
num_idle_blocking_threads
blocking_queue_depth

Not sure if there is much demand for the other two.

[jlizen]

blocking_queue_depth just landed: #49

We'd welcome PRs for the ones that @arielb1 mentioned above, or additional ones if provided with an explanation of use case.

[jlizen]

The other three mentioned above landed as well: #74

Regarding exposing 'all' metric:

I would point out that currently, we only are including 'significant' metrics by design - the structs phrase this as:

Monitors key metrics of the tokio runtime.

and

Monitors key metrics of instrumented tasks.

There is a problem with signal to noise ratio of just passing through everything, this library is a bit more opinionated in that it is trying to craft a useful monitoring/debug story.

Even if we did want to expose another API that includes everything rather than the filtered subset, I have concerns about maintainability. Since, saying it includes 'all' metrics when it doesn't, is perhaps more confusing than just saying it includes 'key' metrics and lazily adding important ones.

Some example issues:

• we target a minimum tokio version, we don't want to bump that without semver breakage and bump people that have older versions pinned, but then only newer versions get new metrics
• there isn't a great way to 'pass through' metrics from tokio without adding eg proc macros on tokio side, which seem unlikely to land, and then current maintainers can't commit to staying on top of new additions manually
• this is further complicated by deciding how to handle this via the new metrics.rs integration to make it easier to flush metrics to exporters. this will require complicating that code with a lot of branching on optional fields, having separate structs, etc.

Practically speaking, we have two new maintainers spinning up (myself and @arielb1), so we have added capacity, but not unlimited. We're open to PRs to add new metrics if you can share a use case for how you are using the metric and why it is generally relevant to consider a 'key' metric. Let's start there.