Implement Asynchronous CSV Export #746
Description
As suggested by @neilh10 under #738, we should consider implementing an asynchronous and event driven download pattern. Moving this to an async process that can run with a longer timeout duration than a typical TCP connection could provide a good scalable pattern to handle downloads for the full range of data as these datasets continue to grow.
I have found a work around - as per an enhanced @ptomasula suggestion, I can download one data point, then when downloaded if I immediately press the "download all" it works. At least for the size of database I'm using so far.
I've since then noticing how other packages deal with the problem - for instance paypal allows older invoices to be accessed - by putting it as batch job and then sending an email
Also using hydrovu.com for Insitu sensors on another project (been amazing so far), on pressing "Export Data" a windowlet pops up saying its working on it, and sends an email. A couple of minutes later I get the email and download from the email link.
As to the bigger picture of solving this, with picking up on any software, characterizing (apparently first detected in Oct 23, 2018, then seems partially fixed in v0.15 ) and identifying the priorities in fixing it - but now I'm still seeing it - pretty close to a show stopper problem.
IMHO its a "show stopper" for recommending MMW, as if you can't download the data, if the systems is expected to scale, and current methods aren't scale-able - what's the point in going to all the pain of installing a wireless network, if failures are likely.
For evolving developments, the project gurus identify the "Valley of Death" in as a potential life cycle phase.
https://www.sciencedirect.com/science/article/pii/S2666188822000119
https://ncinnovation.org/media/how-does-the-valley-of-death-impede-north-carolinas-growth/#:~:text=The%20%E2%80%9Cvalley%20of%20death%E2%80%9D%20is,operate%20in%20real%2Dlife%20settings.
