Use a fixed interval for emitting progress events

[Chaphasilor]

Hey @hgouveia,

I'm currently dealing with a very stubborn server that like to sometimes just stop responding, i.e. stop sending data, without terminating the connection.
When this happens, ndh won't emit any progress events anymore, because they are only emitted when a chunk is received via the tcp socket. Check this line here:

node-downloader-helper/src/index.js

Line 439 in ac8387f

readable.on('data', chunk => this.__calculateStats(chunk.length));

The problem with this is two-fold:

• Because the connection isn't terminated, the state is never updated and it will say in downloading indefinitely, no retries, no errors, nothing
• Because we don't receive any data, the progress isn't updated and the speed, etc. doesn't change at all

=> The download appears to be "frozen".

If we instead used something like setInterval() to regularly emit the stats event, we could avoid this problem.
Firstly, if no new chunk was received since the last interval, we can set the download speed to 0 b/s, or at least decrease it slowly.
And secondly, if no chunks are received for multiple seconds or even a minute, we could emit an error and try to restart the download using the resume feature (if resume is supported, otherwise completely restart it).
This would simply consume one of the available retries, but should allow ndh to recover from such server errors on its own.

What do you think? :)

[hgouveia]

Hello @Chaphasilor , sorry the very big late reply, yeap interesting although i don't like very much the idea of adding a setInterval, i would prefer to keep how it is now, the problem is that sometimes people like to get the information without any throttling, of course these could be fixed by just to have a setInterval(, 0), o very low value, but i am not very happy about it, i will think about it, but what is definitely needed is to have some sort of timeout if not new data is gotten in a period of time, i will take this into account when i do a new release in the future and disccus in here

[Chaphasilor]

That makes sense. How about adding a "watchdog" setInterval that checks the status every minute, and doesn't deal with stats at all, only with updating the state and restarting "frozen" downloads? 🤔

Also, we could combine another setInterval with a sort of timer, so that we keep emitting the stats with each received chunk, but after each interval we check if there was any chunk received during that interval (e.g. 1000ms). If there was no chunk received, we calculate the new stats without needing a chunk, but if there indeed was a chunk, we don't do anything.
This would keep the highest update rate possible, like it is now, but would also work for very slow data rates :)

[hgouveia]

@Chaphasilor yeap i definitely like this approach better, i think we could do this but by default won't do anything and add a new option called watchdog as you suggested, and if true it applies this checking, also emit some event like freeze when detected it so the user could act as they want, but also have a retryOnFreeze option retry automatically

[Chaphasilor]

We could do that, yeah. A lot of new options and behavior though, but it's your call! :D

Once you're decided, I could try to make a PR for this...