[Feature Request]: Add callto: to special_protocols set to prevent errors during crawling

[Eibich]

What needs to be done?

The current implementation of special protocol handling covers mailto:, tel:, ftp:, file:, data:, and javascript:, but does not include callto:. While callto: is not part of any W3C or IETF standard (it was a proprietary scheme introduced by Skype), it is still widely used across many websites. When the crawler encounters a callto: link, it throws an error because the href is not recognized as a special protocol and is presumably processed as a relative URL instead.

crawl4ai/crawl4ai/utils.py

Lines 2444 to 2447 in 1debe5f

	# Handle special protocols
	special_protocols = {"mailto:", "tel:", "ftp:", "file:", "data:", "javascript:"}
	if any(href.lower().startswith(proto) for proto in special_protocols):
	return href.strip()

What problem does this solve?

Make the crawler more resilient against real-world markup.

Target users/beneficiaries

Everyone.

Current alternatives/workarounds

Add a Filter which acceppts only http and https links.

class SchemeFilter(URLFilter):

    _ALLOWED = frozenset({"http", "https"})

    def apply(self, url: str) -> bool:
        try:
            scheme = urlparse(url).scheme.lower()
        except Exception:
            self._update_stats(False)
            return False
        ok = scheme in self._ALLOWED
        self._update_stats(ok)
        return ok

Use the filter where you want to need it:

strategy = BFSDeepCrawlStrategy(
          max_depth=max_depth,
          include_external=False,
          filter_chain=FilterChain([SchemeFilter(), seen_filter, binary_filter]),
          )

Proposed approach

Add callto: to the special_protocols set. Depending on the project's scope, it might also be worth considering other non-standard but commonly encountered URI schemes such as sip:, sips:, skype:, viber:, whatsapp:, or tg: (Telegram) to make the crawler more resilient against real-world markup.

special_protocols = {"mailto:", "tel:", "ftp:", "file:", "data:", "javascript:", "callto:"}