fix: resolve relative URLs in the urllib3 download backend

[ebarkhordar]

focused_crawler() returns nothing for any non-root start URL when trafilatura runs on the urllib3 backend. No redirect is required, which is what makes it reproducible without the site from the report.

Root cause

_send_urllib_request builds the standardised Response from response.geturl():

resp = Response(bytes(data), response.status, response.geturl() or url)

For a PoolManager request, geturl() returns retries.history[-1].redirect_location when a redirect happened, which is the raw Location header and may legally be relative, and otherwise the request URI that was passed down to the connection pool. Both are bare paths. _send_pycurl_request uses curl.getinfo(pycurl.EFFECTIVE_URL), which is always absolute, so the two backends disagree and only the urllib3 one is wrong. That is also why the report could not be reproduced here: with pycurl installed, every case passes.

Downstream, probe_alternative_homepage reads the bare path as a redirect target and assigns it to homepage (spider.py:134-136). get_base_url() then returns '', so the caller guard if htmlstring and homepage and new_base_url is false and focused_crawler yields empty results with no error. The second consumer, process_response at spider.py:232, files the bare path into URL_STORE as visited, so the real URL is never marked.

Fix

urljoin(url, response.geturl() or url). It resolves a relative Location against the requested URL and is a no-op on an absolute one, so it reproduces the pycurl backend's EFFECTIVE_URL semantics and leaves redirect detection intact. Correcting it at the source covers both consumers.

I did not take the response.url not in homepage change suggested in the issue: it patches one of the two call sites, and the reporter noted himself that it breaks redirect handling.

Verification

• New parametrized regression test in tests/downloads_tests.py. On master the two relative cases fail (/news/news, /section/) and the absolute case passes as the no-op control; with the fix all three pass.
• Full suite on 3.13: 342 passed, 1 skipped, plus cli_tests.py::test_sysoutput, which fails identically on unmodified master in the same container because it asserts /root/forbidden/ is unwritable and the container runs as root.
• ruff check ., ruff format --check ., mypy -p trafilatura and python tests/eval_gate.py all pass. The eval gate is unchanged at the pinned floors, as expected for a change that touches no extraction path.
• End to end I compared both backends against a local http.server, across no redirect, relative Location and absolute Location; patched urllib3 matches pycurl in all three. That path is not in the suite, because mock_network in tests/conftest.py replaces _send_urllib_request itself and a spider-level test would mock away the code under test. I ran Python 3.13 and 3.14, not 3.10.

Fixes #696

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.71%. Comparing base (a397f89) to head (6bae98d).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #909   +/-   ##
=======================================
  Coverage   99.71%   99.71%           
=======================================
  Files          21       21           
  Lines        4167     4168    +1     
=======================================
+ Hits         4155     4156    +1     
  Misses         12       12           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.