[question] How to check progress of large downloads?

[wblondel]

Hello everyone,

I'm archiving a website that has a lot of large files, and this takes a very long time.
Is there any way to check the progress of these downloads?
I see files are being downloading by looking at the /tmp folder in the container, but the real name of the files are not there.

As an extra note, I had to set the timeout parameter to 999999, because downloads always timeout after the timeout duration + 100. For example, if timeout is set to 180, I will get Finishing Fetch Timed Out after 280 seconds. I haven't found any settings defaulted to 100.

Current config:

collection: "mycatalog"
workers: 2

seeds:
    - url: https://subdomain.domain.local/
      scopeType: "host"
      extraHops: 1
      exclude:
        - subdomain.domain.local/logout
        - logout$

blockRules:
   - url: googleanalytics.com

profile: /crawls/profiles/profile.tar.gz
generateWACZ: true
combineWARC: true
text: "to-pages"

blockAds: true
timeout: 999999
screencastPort: 9037

[ikreymer]

Are you crawling a site with lots of large static files?
Browsertrix is currently optimized for high-fidelity interactive pages, eg. pages that require Javascript to render, custom page interactions, etc.. Sites that have a lots of large, but static files to download, may not work as well currently - though we'd like to change that! There isn't a way to check pending downloads at the moment, but something we may add in the future.