Coroutine scheduler monitoring

[asad-awadia]

Are there any monitoring tools available for how many coroutines are currently active and their state etc? It would be nice if it could be exposed so that something like Prometheus can scrape it and visualise it in grafana.

It will also help in debugging leaks - and errors that might occur if we see coroutines just rising linearly

If not can this be done by looking at the thread stats instead?

Go exposes it via runtime.NumGoroutines()

Related:? #494

[elizarov]

Please, take a look at kotlinx-coroutines-debug module: https://github.com/Kotlin/kotlinx.coroutines/blob/master/kotlinx-coroutines-debug/README.md

[glasser]

This does look pretty useful, but it also seems like it might have a notable performance impact?

The monitoring that looks attractive to me would be getting a gauge on the sizes of the CoroutineScheduler queues (global and local).

Our biggest fear is accidentally putting slow blocking work (or worse, deadlocks) in our main dispatcher (which happened to us once on a previous project using Kotlin coroutines incorrectly, and also when using Ratpack’s coroutine-style execution).

So getting alerted if work is building up over time (ie, if the queues are getting too big/growing indefinitely) seems helpful.

Would it be reasonable to expose some of these stats somewhere? These stats are specific to the CoroutineScheduler so I don't think kotlinx-coroutines-debug is relevant.

As an awful hack we are considering parsing (Dispatchers.Default as ExecutorCoroutineDispatcher).executor.toString(), with full understanding that it may break at any time.

[elizarov]

The monitoring that looks attractive to me would be getting a gauge on the sizes of the CoroutineScheduler queues (global and local).

@glasser Yes, that can be done without the slow debug mode and makes sense. I'll keep it open as an enhancement.

[glasser]

Thanks! Should I interpret that as "you're going to do it" or "you'd accept patches"?

[qwwdfsad]

Unfortunately, we are not ready to accept patches right now because the scheduler is being actively reworked.

But it would be really helpful if you could provide a more detailed example of the desired API shape and problem you want to solve with this API.

For example, "Ideally, we'd see it as pluggable SPI service for dispatcher with the following methods ..., so we could use to trigger our monitoring if ..."

[glasser]

Interesting — is there a branch or design doc or something for the reworking? Curious how it's changing.

My proposal is pretty simple. A few of the core objects involved with coroutine scheduling should be (a) publicly accessible and (b) expose a few properties that provide statistics about them. It's fine if these are documented as "experiment, up for change, don't rely on this" and as "fetching these properties may have a performance impact if done frequently" (eg, ConcurrentLinkedQueue.size is O(n)).

Most specifically, I'd want to have access to

• ExperimentalCoroutineDispatcher.coroutineScheduler (which perhaps would return an interface declared to only contain the metrics below)
• LimitingDispatcher.queueSize: Int
• CoroutineScheduler.corePoolSize: Int
• CoroutineScheduler.maxPoolSize: Int
• CoroutineScheduler.queueSizes: Map<WorkerState, List<Int>>
• CoroutineScheduler.globalQueueSize: Int
• CoroutineScheduler.schedulerName: String (for tagging in the unlikely case of multiple schedulers)
  (ie, basically all the stuff in CoroutineScheduler.toString(); I think getting the control state isn't super necessary.)

I don't need kotlinx.coroutines to provide any machinery for hooking this up to my metrics service: I'm happy to keep at application (or external library) level the code that takes the dispatchers I care about, polls them for metrics, and publishes to my metrics service of choice.

[qwwdfsad]

Interesting — is there a branch or design doc or something for the reworking? Curious how it's changing.

No for both, though changes will be, of course, properly documented. But mostly it's about changing the parking/spinning strategy without violating liveness property to reduce CPU consumption during the low rate of the requests and to have a robust idle thread termination. Change is just too intrusive and touches all the places in the scheduler.

Thanks for the details!
Could you please clarify, is it for Android app or for some backend service?
Asking because there are also chances that Dispatches.Default will be backed with ForkJoinPool on Android by default (mostly to reduce dex size and count of threads), so we have to interop this observability with FJP as well.

[glasser]

This is for server usage.

We are currently porting a few web servers from Ratpack to Ktor. Ratpack has a similar async structure (with a recommended usage of a pool of "compute" threads approximately equal in size to the number of CPUs plus a scaling "blocking" pool) to Kotlin coroutines, but because you have to do all work with explicit Promise composition rather than the nice syntax of Kotlin coroutines, we've found that developers often don't bother to keep blocking work out of the compute pool, and often implement error handling incorrectly (eg by putting try/catch/finally or retry loops around functions that return Promises rather than properly using the Promise API). Our hope is that Kotlin coroutines will be much more accessible. But we still want to monitor that we're not clogging up the pools!

(Ratpack Promises also have some other odd behavior — eg, Blocking.get {}, which is somewhat like withContext(Dispatchers.IO {}, does not actually invoke the given block on the scalable threadpool until after the currently-running code fully returns to the event loop (equivalent of suspension), which meant that some misguided attempts to make a blocking call within a non-Promise-returning function use the "right" threadpool by writing (effectively) Blocking.get {}.get() not only tied up the current thread like you might expect, but actually blocked indefinitely because the code never got run! Hopefully our complete rewrite will avoid these border cases.)