Home
x-crawl is a flexible nodejs crawler library. It can crawl pages in batches, network requests in batches, download file resources in batches, polling and crawling, etc. Flexible and simple to use, friendly to JS/TS developers.
If you like x-crawl, you can give [x-crawl repository](https://github.com/coder-hxl/x-crawl) a star to support it, not only for its recognition, but also for Approved by the developer.
- 🔥 Async/Sync - Just change the mode property to toggle async/sync crawling mode.
- ⚙️ Multiple functions - Batch crawling of pages, batch network requests, batch download of file resources, polling crawling, etc.
- 🖋️ Flexible writing method - A function adapts to multiple crawling configurations and obtains crawling results. The writing method is very flexible.
- ⏱️ Interval crawling - no interval/fixed interval/random interval, can effectively use/avoid high concurrent crawling.
- 🔄 Retry on failure - It can be set for all crawling requests, for a single crawling request, and for a single request to set a failed retry.
- 🚀 Priority Queue - Use priority crawling based on the priority of individual requests.
- ☁️ Crawl SPA - Batch crawl SPA (Single Page Application) to generate pre-rendered content (ie "SSR" (Server Side Rendering)).
- ⚒️ Controlling Pages - Headless browsers can submit forms, keystrokes, event actions, generate screenshots of pages, etc.
- 🧾 Capture Record - Capture and record the crawled results, and highlight them on the console.
- 🦾TypeScript - Own types, implement complete types through generics.