Context
Benchmarking the bulk download engine against wget + GNU parallel showed ia is ~22% slower at 4 workers (gap narrows to ~2% at 8 workers when the connection saturates). Root cause is per-request overhead that prevents HTTP connection reuse.
Benchmark details: docs/bulk-download-performance.md (on bulk-engine branch).
Changes
1. Remove Connection: close header — session.py:141
self.headers.update({'Connection': 'close'})Forces a new TCP+TLS handshake for every HTTP request. Removing it lets urllib3's connection pooling work as designed — each worker reuses a single connection per host instead of opening one per file.
Risk: May have been added to work around a server-side issue at archive.org. Test against a large item (1000+ files) and monitor for 503s or connection resets. Safer first step: only override in the bulk download path via DownloadWorker._get_session().
2. Skip redundant mount_http_adapter() per file — files.py:225
self.item.session.mount_http_adapter(max_retries=retries)
Called at the start of every File.download(). Creates a new HTTPAdapter with a fresh connection pool, discarding existing connections. Only needs to run if the retry config actually changed.
3. (Lower priority) Avoid deepcopy in get_files() — item.py:682
deepcopy(self.files) runs on every get_files() call but is only needed when on_the_fly=True. A shallow copy or iterator chain is sufficient.
Related
Expected Impact
Items 1 + 2 together should close most of the throughput gap with wget at low worker counts (~10–30% improvement). Item 3 is marginal except for items with thousands of files.
Context
Benchmarking the bulk download engine against
wget+ GNUparallelshowediais ~22% slower at 4 workers (gap narrows to ~2% at 8 workers when the connection saturates). Root cause is per-request overhead that prevents HTTP connection reuse.Benchmark details:
docs/bulk-download-performance.md(onbulk-enginebranch).Changes
1. Remove
Connection: closeheader —session.py:141Forces a new TCP+TLS handshake for every HTTP request. Removing it lets urllib3's connection pooling work as designed — each worker reuses a single connection per host instead of opening one per file.
Risk: May have been added to work around a server-side issue at archive.org. Test against a large item (1000+ files) and monitor for 503s or connection resets. Safer first step: only override in the bulk download path via
DownloadWorker._get_session().2. Skip redundant
mount_http_adapter()per file —files.py:225Called at the start of every
File.download(). Creates a newHTTPAdapterwith a fresh connection pool, discarding existing connections. Only needs to run if the retry config actually changed.3. (Lower priority) Avoid
deepcopyinget_files()—item.py:682deepcopy(self.files)runs on everyget_files()call but is only needed whenon_the_fly=True. A shallow copy or iterator chain is sufficient.Related
Expected Impact
Items 1 + 2 together should close most of the throughput gap with wget at low worker counts (~10–30% improvement). Item 3 is marginal except for items with thousands of files.