How can i send the request to fetch data from external api rather than launching a browser locally ?

[ardhrubo]

Hi, I am working with Crawlee and want to retrieve data from an external API without launching a browser.

I want to send a request to the external source, fetch the data, and then process it using the Cheerio crawler. Using Crawlee’s BasicCrawler, I can fetch data from external sources, but I am facing an issue with the built-in URL strategy.

While BasicCrawler allows me to fetch data, the default enqueuing strategy doesn't seem to allow crawling the same hostname or subdomains the way Cheerio and Puppeteer crawlers do.

This is causing a problem when trying to process the fetched URLs that belong to the same or different subdomains.

Is there a way to enable BasicCrawler to crawl the same hostname or subdomain, similar to the behavior provided by Cheerio or Puppeteer crawlers?

Any advice or workaround to overcome this limitation would be very helpful.

[janbuchar]

Hi @ardhrubo and thanks for your interest in Crawlee. Could you send a code snippet that illustrates the problem you're facing? I have a hard time understanding how enqueueLinks fits in with the rest of your issue.

[ardhrubo]

Yes, you are correct.

I used crawlee.dev for testing purpose but it wasn't working.Then also tried with example.com and tried using my locally setup website. Also tried with python http server.

It works fine with cheerio crawler. But i can not use it with basic crawler. Here is an example function which is not possible.

const crawlHostname = async (c: Context) => {
  const { url, format = 'json' } = await c.req.json();
  if (!url) {
    return c.json({ error: 'url is required' }, 400);
  }

  const crawler = new BasicCrawler({
    async requestHandler({ request, log, pushData }) {
      const { url } = request;
      log.info(`Processing ${url}...`);

      try {
        const data = await fetchdata(url);
        await pushData({ url, data });
      // crawl all the hostname
        await enqueueLinks({ strategy: EnqueueStrategy.SameHostname });

        log.info(`Successfully processed ${url}`);
      } catch (error) {
        log.error(`Failed to process ${url}: ${error.message}`);
      }
    },
    maxRequestsPerCrawl: 10,
    maxConcurrency: 10,
  });

  await crawler.addRequests([url]);
  await crawler.run();

  const dataset = await Dataset.getData();
  return formatResponse(c, dataset.items, format);
};
 

[janbuchar]

I see two potential issues here.

1. async requestHandler({ request, log, pushData }) { - on this line, you should also take enqueueLinks from the crawling context via destructuring. In your snippet, it is undefined, but I suspect that's due to you trimming down the example, correct?
2. await enqueueLinks({ strategy: EnqueueStrategy.SameHostname }); in BasicCrawler, the enqueueLinks handler requires you to pass the URLs to enqueue, because it does not assume anything about the structure of the page. CheerioCrawler expects the page to be HTML, and so it looks for a elements instead of relying on the caller to pass the URLs in. So, the function call in BasicCrawler should look something like this - await enqueueLinks({ urls: ["https://1.com", "https://2.com"], strategy: EnqueueStrategy.SameHostname });

[ardhrubo]

Oh I see, I have to pass the urls as you mentioned. But It would have been better for me if I can do that like cheerio or puppeteer crawler.

But anyway, thank you so much for helping me.

[janbuchar]

The problem is that BasicCrawler does not use an HTML parser implicitly, so it doesn't have a way to find the links automatically. We will at least try to document this better 🙂

[ardhrubo]

Yeah, That would have been more helpful if it was in the documentation.