Impit panics with "No route to host" and causes retries / RustPanic

[vdusek]

Reproduction - ParselCrawler with ImpitHttpClient:

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.http_clients import ImpitHttpClient
from crawlee.storage_clients import MemoryStorageClient


async def main() -> None:
    storage_client = MemoryStorageClient()
    http_client = ImpitHttpClient()

    crawler = ParselCrawler(
        storage_client=storage_client,
        http_client=http_client,
    )

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing URL: {context.request.url}...')
        data = {
            'url': context.request.url,
            'title': context.selector.css('title::text').get(),
        }
        await context.push_data(data)
        await context.enqueue_links()

    await crawler.run(['http://crawlee.dev/'])


if __name__ == '__main__':
    asyncio.run(main())

While running the exact same example with HttpxHttpClient works reliably:

import asyncio

from crawlee.crawlers import ParselCrawler, ParselCrawlingContext
from crawlee.http_clients import HttpxHttpClient
from crawlee.storage_clients import MemoryStorageClient


async def main() -> None:
    storage_client = MemoryStorageClient()
    http_client = HttpxHttpClient()

    crawler = ParselCrawler(
        storage_client=storage_client,
        http_client=http_client,
    )

    @crawler.router.default_handler
    async def request_handler(context: ParselCrawlingContext) -> None:
        context.log.info(f'Processing URL: {context.request.url}...')
        data = {
            'url': context.request.url,
            'title': context.selector.css('title::text').get(),
        }
        await context.push_data(data)
        await context.enqueue_links()

    await crawler.run(['http://crawlee.dev/'])


if __name__ == '__main__':
    asyncio.run(main())

I ran both versions multiple times to confirm the difference.

At first, pages are processed normally:

...
[ParselCrawler] INFO  Processing URL: http://crawlee.dev/js/api/next/utils/function/isLambda...
[ParselCrawler] INFO  Processing URL: http://crawlee.dev/js/api/next/utils/function/getMemoryInfo...
[ParselCrawler] INFO  Processing URL: http://crawlee.dev/js/api/next/utils/function/htmlToText...
[ParselCrawler] INFO  Processing URL: http://crawlee.dev/js/api/next/utils/function/gotScraping...
...

Then the client starts throwing internal errors, triggering retries:

[ParselCrawler] WARN  Retrying request to http://crawlee.dev/js/api/next/utils/interface/OpenGraphProperty due to:
The internal HTTP library has thrown an error:
  File ".../crawlee/http_clients/_impit.py", line 132, in crawl
    response = await client.request(...)

Eventually, it escalates into a Rust panic:

[ParselCrawler] WARN  Retrying request to http://crawlee.dev/js/docs/3.12/guides/got-scraping due to:
rust future panicked: unknown error

[ParselCrawler] ERROR Request to https://crawlee.dev/js/api/puppeteer-crawler/interface/PuppeteerCrawlerOptions failed and reached maximum retries
Traceback (most recent call last):
  ...
  File ".../crawlee/http_clients/_impit.py", line 132, in crawl
    response = await client.request(...)
pyo3_async_runtimes.RustPanic: rust future panicked: unknown error

So the crawler works fine for a while, then suddenly requests start failing with HostUnreachable / rust future panicked: unknown error. With HttpxHttpClient, this never happens.

Full logs:

• crawler_httpx.log
• crawler_impit.log

[barjin]

Closed by apify/impit#255 , please update to impit@0.6.0 .