[Bug]: There is no way to download large files

[AramZS]

Browsertrix Version

v1.21.0-2793d2c

What did you expect to happen? What happened instead?

I have a few smaller crawls that are between 1-3 gigs that appear to be downloading fine, but my crawls that are above 10gigs are impossible to download from the web UI. They seem to just fail out given enough time with the server just no longer responding with additional file data. I'm fine using other methodology but none seem to be clear and I don't want to delete my archives just so my account can start working again (it is full).

Reproduction instructions

• Run a crawl that collects larger than 10 gigs worth of data
• Attempt to download it.

This was one of my crawl lists:

Some of these sites are down now, so I'm not sure you'll get the same results.

https://www.train.org/cdctrain/course/1117398/details
https://www.mdpi.com/1660-4601/11/6/6433
https://www.apha.org/topics-and-issues/climate-health-and-equity/jedi
https://www.cdc.gov/climateandhealth/climate_ready.htm
https://www.apha.org/topics-and-issues/racial-equity/racism-declarations
https://www.pathlms.com/health/courses/64574
https://ephtracking.cdc.gov/Applications/HeatRisk/
https://www.ready.gov/
https://www.cdc.gov/natural-disasters/
https://endingracism.apha.org/
https://nca2023.globalchange.gov/
https://atlas.globalchange.gov/#data
https://atlas.globalchange.gov/#explore
https://nca2018.globalchange.gov/
https://health2016.globalchange.gov/
https://science.nasa.gov/climate-change/

Not run locally.

Screenshots / Video

Other settings:

Environment

OSX Tahoma, Arc (Chrome) latest.

Additional details

No response

[ikreymer]

Sorry to hear you're having issues downloading from Browsertrix. Are you using the 'Download All' link or the individual WACZ file downloads that are listed? The individual downloads should be directly from S3-storage provider, and should download fine, while the 'Download All' link combines the files on the fly into one, and that's happening on our backend and perhaps that ends up timing out for you? The individual WACZ files should all be in ~5GB in size, as we split them up into files of that size.

[AramZS]

Ah, I was trying to download them from the Archived Items page from the items menu. I didn't realize that page existed. It did work when I went to it! It might be useful as a feature to redirect the download link for items that are too large to that inner-item page or have some sort of mouse over notice. Thank you, got it working now!

[AramZS]

@ikreymer I did some testing and it looks like crawls can generate files in excess of 6gigs and once they break that threshold they become impossible to download. It seems the download fails around the 5 gig mark.

If it helps:

Workflow ID: dc792103-807b-47d9-ab5c-6a521c94642e
ID: manual-20250305011130-dc792103-807

[ikreymer]

Hm, I was able to download the 6GB+ locally now, also cleared the CDN cached and tweaked the setting a bit. Can you try again to see if it works better?

Also, are you downloading via the browser? Would try via curl / wget in case that works better also.