Interrupted `balena.logs.subscribe` callbacks

[pranasziaukas]

It was brought to our attention by a customer that balena.logs.subscribe callbacks cease happen in some cases.
The code sample seems to be barebones except for the fact that it loops balena.models.device.get_all_by_application_id or balena.models.device.get_all to subscribe to new devices.

I was able to reproduce it running both python 3.8.2 (same as customer) and the latest 3.9.1 although Marios does not see the same issues.

I thought that it could be the time.sleep somehow messing with the async process but after replacing it with the async-friendly twisted.internet.task.LoopingCall the issue seems to persist.
Given such circumstances, it is in line with balena.models.device.get_all_by_application_id or balena.models.device.get_all somehow interrupting the callbacks as without them balena.logs.subscribe keeps on running without issues.

Further context and code samples can be found in the attached support ticket. Maybe you could assess the situation @nghiant2710. Thanks!

[jellyfish-bot]

[pranasziaukas] This issue has attached support thread https://jel.ly.fish/8ff47709-93a5-40e7-9797-88d1a7ddea48

[nghiant2710]

thanks @pranasziaukas, will have a look into this and get back to user when I have more info.