WARC-Resource-Type field possibilities (feedback wanted)

[ikreymer]

Browsers have different ways of reporting the 'resource type' for any resource that's being fetched. When using browser-based crawling, it is often easy to access this 'resource type' and store it in a custom WARC header.

It is possible to introduce a WARC-Resource-Type header to store this type. Unfortunately, there isn't a single standard of 'resource types' and various browser APIs expose different variations on this.

If a resource type is written to a WARC header, is there a way to make it future proof to support different vocabularies?

Some possibilities include:

• Chrome Debug Protocol (CDP) resource type
  this is easiest for Chromium-browser based crawling as these fields are directly accessible, but is not especially well standardized and could change anytime.

• Fetch Request.destination - this is well standardized vocabulary but not a one-to-one mapping and may not be accessible for non-Fetch data.

• Extension API webRequest.resourceType - better standardized and supported by all the major browsers with some differences for browser extensions. Not quite one-to-one with CDP types.

One approach to make this more future proof might be to prefix the resourceType with a namespace based on where the data is coming from and which vocabulary is used.

For example, if using CDP, cdp:Document or cdp:Image, if using webRequest, might be webRequest:sub_frame, webRequest:image, if using destination, destination:image, destination:document, etc...

This allows for expanding into other vocabularies in the future, but may be harder to parse.

Alternatively, there could be a fixed vocabulary that is allowed that is a common subset of at least 2 of the above, which might be:
document, image, media, script, stylesheet, font, ping, websocket, fetch and a catch-all other.

(In this case, we should specify what the more specific values are recorded as, eg. main_frame / sub_frame would be recorded as document)

Other thoughts / suggestions welcome!

[ikreymer]

I should note our initial implementation just stores the Chrome CDP value, eg. WARC-Resource-Type: Document, WARC-Resource-Type: Image, etc... w/o a prefix, as that was the easiest to try. We could also just keep that, but wanted to see if there were any thoughts on the above proposals. Other tools that work directly with Chrome Debug Protocol, such as Brozzler or the Chrome Extractor for Heritrix, would actually have the same vocabulary as well, so may not be an immediate concern.
Mostly a question of other tools / future proofing to support vocabulary not coming from CDP, if such a header were to be standardized.

[tw4l]

Note that Puppeteer and Playwright use the CDP values but lowercased: https://playwright.dev/docs/api/class-request#:~:text=resourceType%E2%80%8B&text=ResourceType%20will%20be%20one%20of,%2C%20websocket%20%2C%20manifest%20%2C%20other%20

[tw4l]

Playwright mapping for Firefox: https://github.com/microsoft/playwright/blob/73ffaf65d75b2378168ac5a11eb37cced03ff6ea/packages/playwright-core/src/server/firefox/ffNetworkManager.ts#L161

[ato]

Do we have any use cases in mind for this field when reading the WARC?

I guess one might be be listing all the top-level crawled documents. This can't be done accurately by Content-Type alone as XHR/Fetch requests can have text/html responses.

The main_frame/sub_frame distinction also seems interesting for that use case. It's not in the CDP resource type but if we map to one of the other vocabularies presumably it could be determined from the frameId?

I guess the hopsFromSeed metadata field could be used for listing top-level crawled documents but it's coarse grained and doesn't make distinctions between different kinds of embedded content.

It's also possible for an image to have a text/html Content-Type and still display correctly due to MIME sniffing. So similarly if you wanted to do something with all the images in a crawl, Content-Type alone is insufficient.

[tw4l]

We've added this to our WARCs in response to a user-submitted issue: webrecorder/browsertrix-crawler#451, with the primary use case being differentiating between resources fetched by JavaScript (via fetch, xhr) versus resources loaded directly from the HTML.

[edsu]

This is probably off topic for this issue, but it came up recently in the context of using mailbagit that it would be useful to know if a record is for a seed URL. Or is there another common way of doing that? The motivation here is to be able to pick out URLs from the WARC data to serve as entry points during replay.

[ato]

it would be useful to know if a record is for a seed URL. Or is there another common way of doing that?

For WARCs created by Heritrix a metadata record without the via and hopsFromSeed fields is indicative of a seed. If the crawler doesn't populate those fields though I don't think there's a reliable way to tell from a WARC file alone. Requests without a Referer header might also be indicative for some crawlers but but not ones that obey Referrer-Policy: no-referrer.

WACZ defines an accompanying pages.jsonl file for entry points.

[benoit74]

See webrecorder/browsertrix-crawler#630 for a feedback "from the trenches".

[Mr0grog]

I guess one might belisting all the top-level crawled documents.

This is the exact use case I was trying to solve today when I found this issue. I’m looking at a Browsertrix.crawler WARC and having the CDP resource type in WARC-Resource-Type is helpful (thanks @tw4l / @ikreymer!), but agree with others this doesn’t quite solve that use-case alone since the contents of iframes will also have the document type.

The main_frame/sub_frame distinction also seems interesting for that use case. It's not in the CDP resource type but if we map to one of the other vocabularies presumably it could be determined from the frameId?

This would be great (for my use case, at least 🙂). For Puppeteer-based tools, this is definitely get-able; the easiest is probably looking at the Request object’s .frame().parentFrame() to determine if main frame.

---

CDP’s Initiator might be another way to help solve this. In Chrome, it’s other for top-level things like the document and favicon, so I think you could only have one thing with ResourceType = Document && Initiator = other. But I could definitely be wrong there. Unfortunately, Firefox looks like it treats Initiator very differently (just glancing at the devtools UI; have not looked at any code), although it’s closer to the frame/subframe idea with values like document and subdocument.

I know the new WebDriver-BiDi spec is intended to supplant CDP for tools like Selenium/Puppeteer/Playwright, so it might be worth keeping an eye on that. Firefox has already deprecated its CDP implementation in favor of it, although it’s still under pretty active revision. Anyway BiDi RequestData objects have:

• destination, which corresponds to destination from the fetch spec, so I think in BiDi this should be available on all requests 🤞
• initiatorType, which corresponds to initiator type from the fetch spec. (Note there is a different Initiator type in the BiDi spec that is deprecated. There’s an open issue about clarifying it. So this is messy. 🙁)
• context, which is an ID for the navigable (frame) the request is happening in. So I think this would be the correct way to get main frame vs. sub-frame in BiDi.