Thousands of tokio::task <cargo>/tokio-1.15.0/src/task/local.rs:302:22 lost their waker

[Antiarchitect]

I don't actually know how to report this. But my app uses lapin and mqtt-proxy crates which are run on tokio runtimes. After integrating with tokio-console I see multiple (thousands in couple of minutes) tasks which points here: https://github.com/tokio-rs/tokio/blob/tokio-1.15.0/tokio/src/task/local.rs#L302 After some time I disconnect MQTT client and the app fails:

thread 'ntex-rt:worker:1' has overflowed its stack
fatal runtime error: stack overflow

This happens only when console-subscriber is initialized (or maybe I wait not long enough). If you have any ideas in order how to diagnose it further - let me know.

P.S. The tool is awesome BTW - this exact tool I missed during my async apps development. Hope it will get better and feature-rich!

[hawkw]

My guess is that the very large number of lost waker warnings is due to issue #149 --- warning sometimes causes false positives, which we're working on fixing. Sorry about that!

I'm not sure about the stack overflow, but it's certainly possible that's also related to the console instrumentation --- are you running in release mode?

[Antiarchitect]

@hawkw Caught this error in debug mode. Not sure about the release - will try to reproduce it.

[Antiarchitect]

I see this bug in the release build also. https://i.imgur.com/cC2BHAK.png

[hawkw]

I see this bug in the release build also. https://i.imgur.com/cC2BHAK.png

Yes, the false positives for the lost waker warning is expected to occur in both debug and release builds. I was wondering if the stack overflow in the application can be reproduced in release or debug mode.

[Antiarchitect]

@hawkw couldn't wait that long. Can try to perform some long run for that.

[hawkw]

I believe #238 fixes this issue. Are you able to try the latest git revision of console-subscriber (fdc77e2) and confirm that this is the case? You'll also need to build the git version of the console CLI.

[Antiarchitect]

Will check a bit later

[Antiarchitect]

Sorry for the delay https://i.imgur.com/MGLHdVz.png
It's strange that now console shows some highly busy tasks as lost their waker (I see busy time increases) and multiple polls but Waker stats are empty https://i.imgur.com/tsgfLj0.png how could it be?

P.S. I use fdc77e28d45da73595320fab8ce56f982c562bb6 for both tokio-console and console-subscriber