Crawl visited Domain out of Scope

[gitreich]

This Issue has the following impact:

A domain which was not in the scope of the crawl was found in the WARC File (Request & Response)
The false domain was andererseits.org
the scope of the crawl shoud have only be materie.at

But this Domain was not found in:
docker logs (or log file in sub directory logs)
page.jsonl
extraPages.jsonl
crawl-state.yaml (from sub folder crawl)

It was reproduceable with Version 1.5.4 with the following command:

docker run -d --name ONB_Btrix_materie_politik_daily_20250305014538 -e NODE_OPTIONS='--max-old-space-size=32768' -p 39833:39833 -p 38665:38665 -v /home/netarchive/browsertrix/crawls/:/crawls/ webrecorder/browsertrix-crawler:1.5.4 crawl --screencastPort 39833 --healthCheckPort 38665 --scopeType domain --headless --delay 1 --behaviorTimeout 120 --pageLoadTimeout 120 --waitUntil networkidle0 --saveState always --logging stats,info --config /crawls/config/materie_politik_daily_20250305014538.yaml --depth 1 --workers 1 --limit 2500 --sizeLimit 1181116007 --timeLimit 43200 --collection materie_politik_daily

(some params are removed from our live crawl)

Here is the config.yaml:

seeds:

• url: http://materie.at/
  scopeType: "prefix"
  depth: 3
  exclude:
  • googlevideo.com
• url: http://materie.at/impressum/
  scopeType: "page"
  depth: 0
  worker: 1

And here is the link in the html of materie.at:

<p>Folge 35: <a href="https://andererseits.org/" data-type="link" data-id="https://andererseits.org/" target="_blank" rel="noreferrer noopener">Andererseits</a> ist ein Magazin für Behinderung und Gesellschaft, bei dem Journalistinnen und Journalisten mit und ohne Behinderung „gleichberechtigt, kritisch und fair bezahlt“ arbeiten. Man könnte und sollte auch „investigativ“ hinzufügen. Die erste Recherche mit breiter Aufmerksamkeit richtete sich gegen die problembehaftete ORF-Sammelaktion „Licht ins Dunkel“. Vor Kurzem entstand gemeinsam mit dem Investigativmedium „Dossier“ und dem Karikaturisten Gerhard Haderer eine (gedruckte) Ausgabe über Behinderungen am österreichischen Arbeitsmarkt.</p>
<p>Der Mitgründer und Co-Geschäftsführer von andererseits Lukas Burnar erörtert im Gespräch mit Niko Alm Hintergründe zu diesem Projekt und das Wirken von Andererseits.</p>

Here is the response header of the warc:

WARC/1.1^M
WARC-Page-ID: 64edc5bb-023b-4d24-8d1c-73dd74dbaece^M
WARC-Target-URI: https://andererseits.org/^M
WARC-Date: 2025-03-07T13:17:30.998Z^M
WARC-Type: response^M
WARC-Record-ID: urn:uuid:1da53545-e000-4a15-afbc-fa2c433b618f^M
Content-Type: application/http; msgtype=response^M
WARC-Payload-Digest: sha256:f78732cd573961b7ec798d60330f4d005e64f0137e91589f4fafe58e34cb381f^M
WARC-Block-Digest: sha256:2976593ae04f278fe781f8aa28cea59e7fe3aaf6f8fad4ec899798ecb9e6ac3a^M
Content-Length: 1080503^M

The response itself is huge so i skip it here

The Request in the WARC File:

WARC/1.1^M
WARC-Concurrent-To: urn:uuid:1da53545-e000-4a15-afbc-fa2c433b618f^M
WARC-Page-ID: 64edc5bb-023b-4d24-8d1c-73dd74dbaece^M
WARC-Target-URI: https://andererseits.org/^M
WARC-Date: 2025-03-07T13:17:30.998Z^M
WARC-Type: request^M
WARC-Record-ID: urn:uuid:1529a409-a381-43ce-89c7-b3b8b7c7bfff^M
Content-Type: application/http; msgtype=request^M
WARC-Payload-Digest: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855^M
WARC-Block-Digest: sha256:8ac8a2c19d1c441af9f62e3e2309d3adaf82ddc6cbc8f100770cc234df048e17^M
Content-Length: 18^M
^M
GET / HTTP/1.1^M
^M
^M
^M

This request should never have been generated with the scope domain of docker start or prefix from the config.yaml

[ikreymer]

This URL is treated not as a page or a link, but as an extra URL that is fetched dynamically, through the Autofetch behavior, because it is listed in the data-* attribute. This logic adds all URLs that are in data-* attributes, I believe
https://github.com/webrecorder/browsertrix-behaviors/blob/main/src/autofetcher.ts#L334

Perhaps this is too aggressive, and it should be excludable with --blockRules andererseits, though I think this default makes sense, as data-* often contain important URLs that should be captured.

[gitreich]

hmm okay, this is not the only domain related to the issue, so there would emerge more than one block rule out of this issue. But can at least the wrong domain can be listed in extra_page.json? This would make it much easier to investigate if a crawl breaks out of scope (manually check every page on data-* attributes on the website seems like a huge overhead to me)

[ikreymer]

You can also disable autofetch if it is not needed for a particular crawl. I suppose we can add options for customizing autofetch also but that gets more complicated.

But can at least the wrong domain can be listed in extra_page.json?

Well, its not a page, its just an extra resource as far as browser is concerned - it's just fetched like await fetch("https://andererseits.org/") - its never loaded in the browser directly.

Perhaps data-type="link" values can be excluded, but that's not really standard. Or, can check if its actually html, and then skip it (though that could not be correct) Open to other suggestions.

[gitreich]

I wanted to know if blockRule can really prevent the crawler of writing the andererseits response into the WARC
I setted up the test with the following config:

seeds:
- url: https://materie.at/podcasts/
  scopeType: "prefix"
  depth: 1
  waitUntil: "domcontentloaded"
  exclude:
     - googlevideo.com     
blockRules:
- andererseits

(I also tried it with exclude, the result was the same)
The request and responses has been like in the initial Issue post:

WARC/1.1^M
WARC-Page-ID: 833bf9f7-6dad-4d55-9460-7b5cfd4472d4^M
WARC-Target-URI: https://andererseits.org/^M
WARC-Date: 2025-04-03T10:04:13.128Z^M
WARC-Type: response^M
WARC-Record-ID: urn:uuid:a06c71d0-cb8a-40f9-b5ff-fad36fe490e8^M
Content-Type: application/http; msgtype=response^M
WARC-Payload-Digest: sha256:ec1fa7785a02565308e782a71913916e3623c12577f4bf0f3c9625d589641567^M
WARC-Block-Digest: sha256:cf18fb7fb7bf3dd9513884a8d1879b347093ec2d7578025d19aceec7654b0aed^M
Content-Length: 1080657^M
^M
HTTP/1.1 200 OK^M