Problem generating downloads of large event files

[IgorA100]

For example, we need to generate a 60GB file, but we only have 40GB of disk space.
In this case, Apache will take up all the disk space, causing a problem. But we won't know about it because the Frontend will timeout (unless, of course, the timeout is set to 10 minutes or more).
The temporary file (or rather, TWO huge ones!) will remain in the system until you manually delete them!

The second problem: We have 40GB of disk space, and we want to generate a 30GB file. But this will also fail because the file will be archived and will eventually take up ~60GB, which we don't have...

Yes, with infinite disk space and the Frontend timeout set to infinity, there won't be any problems, but in reality, things are different...

I don't yet know how to quickly fix this problem.
I see something like this:

• Analyze disk space before file generation and notify the user if space is low. - Offer a download option without archiving (archiving videos is pointless, and you can always sacrifice the text file included in the archive for space)
• Run the generation and download process in the background to avoid Frontend timeouts

[connortechnology]

We already estimate the size of the download based on a sum of event disk space. So it should be trivial to provide a warning.

[IgorA100]

We already estimate the size of the download based on a sum of event disk space.

Yes, that's right.
But all this needs to be done in the background so it doesn't depend on Frontend timeouts and add the ability to download unzipped files.
When I have time, I'll probably come up with something.

[IgorA100]

After downloading the generated archive, all files (generated mp4, txt, zip) remain on my server until I restart the server.
Tested in Chrome & Chromium
I don't think this is a good idea.

[connortechnology]

Hard to know when to delete, which is why we let tmpfiles.d auto-delete them at the end of the day. Still a potential DOS though.

I don't actually see why this was done in a two-step way, step 1 create the file return the file name, step 2 browser requests the file. If we did it all in one request, we could delete it at the end of the request.

[IgorA100]

Yes, I also have questions about generating and downloading files.
I'll think about the best way to do it.

[IgorA100]

I'm working on this problem. A lot of things need to be redone.
Firstly, files will be added to the archive one by one. That is, an MP4 file is generated and immediately added to either a ZIP or TAR file. This significantly saves disk space for intermediate files.
Second, it will be possible to download files without archiving. Links will be generated and displayed to the user.
Third, all intermediate temporary files will be deleted as the archive is created.
Fourth, downloaded files will also be deleted automatically.
And so on.

[IgorA100]

@connortechnology

I can't solve the following problems yet:

• We generated several files for downloading and sent the links to the browser, but for some reason the user didn't download one or more of the files. Perhaps they changed their mind about downloading, or there might be a network issue. What should I do?

We can monitor the loss of connection to the browser via AJAX requests. If the connection is lost, then we can delete the generated files. But if the connection isn't lost, what should I do? How long should I wait? Perhaps the user is downloading, but the network is slow and the download will take a very long time. And here's another problem: Let's say the user started downloading a file and closed the browser page. In this case, AJAX requests will stop working and it will be assumed the connection is lost, but this is not the case. What should I do?

I can't solve these problems yet.

[connortechnology]

The event files may be stored on a slow medium (Amazon S3) etc. The generation of the download may be VERY slow. We can't really get around this, which is why we do it this way.

We now have a file browsing view so they could come back later and check to see if the download is available.
We can add a notification system so that when they come back they get a popup.

[IgorA100]

I am not closing this Issue, because... the work is not 100% done yet.
I'll finish with the audio visualization and come back here...