Skip to content

Home

Jump to bottom
CoderHXL edited this page Mar 10, 2023 · 5 revisions

x-crawl

x-crawl is a flexible nodejs crawler library. Used to crawl pages, batch network requests, and batch download file resources. Crawl data in asynchronous or synchronous mode, 3 ways to get results, and 5 ways to write requestConfig. Runs on nodejs, friendly to JS/TS developers.

If you feel good, you can support x-crawl repository with a Star.

Features

  • Cules data for asynchronous/synchronous ways.
  • In three ways to obtain the results of the three ways of supporting Promise, Callback, and Promise + Callback.
  • RquestConfig has 5 ways of writing.
  • The anthropomorphic request interval time.
  • In a simple configuration, you can capture pages, JSON, file resources, and so on.
  • The rotation function, crawl regularly.
  • The built -in Puppeteer crawl the page and uses the JSDOM library to analyze the page, or it can also be parsed by itself.
  • Chopening with TypeScript, possessing type prompts, and providing generic types.
Clone this wiki locally