Update: Internal type adjustments to catch errors in crawlPage API

Author: coder-hxl

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+  "typescript.tsdk": "node_modules\\typescript\\lib"
+}

README.md

@@ -9,12 +9,12 @@ If you feel good, you can give [x-crawl repository](https://github.com/coder-hxl
 ## Features
 
 - Support asynchronous/synchronous way to crawl data.
-- The writing method is very flexible and supports multiple ways to write request configuration and obtain crawling results.
+- Flexible writing, support a variety of ways to write request configuration and obtain crawl results.
 - Flexible crawling interval, up to you to use/avoid high concurrent crawling.
 - With simple configuration, operations such as crawling pages, batch network requests, and batch download of file resources can be performed.
 - Possess polling function to crawl data regularly.
 - The built-in puppeteer crawls the page, and uses the jsdom library to analyze the content of the page, and also supports self-analysis.
-- Capture and record the success and failure of batch crawling, and highlight the reminders.
+- Capture the success and failure of the climb and highlight the reminder.
 - Written in TypeScript, has types, provides generics.
 
 ## Relationship with puppeteer 
@@ -65,21 +65,20 @@ The return value of the crawlPage API will be able to do the following:
 - [Types](#Types)
     * [AnyObject](#AnyObject)
     * [Method](#Method)
-    * [RequestConfigObject](#RequestConfigObject)
+    * [RequestConfigObjectV1](#RequestConfigObjectV1)
+    * [RequestConfigObjectV2](#RequestConfigObjectV2)
     * [RequestConfig](#RequestConfig)
-    * [MergeRequestConfigObject](#MergeRequestConfigObject)
     * [IntervalTime](#IntervalTime)
     * [XCrawlBaseConfig](#XCrawlBaseConfig)
-    * [CrawlBaseConfigV1](#CrawlBaseConfigV1)
     * [CrawlPageConfig](#CrawlPageConfig	)
+    * [CrawlBaseConfigV1](#CrawlBaseConfigV1)
     * [CrawlDataConfig](#CrawlDataConfig) 
     * [CrawlFileConfig](#CrawlFileConfig)
     * [StartPollingConfig](#StartPollingConfig)
-    * [XCrawlInstance](#XCrawlInstance)
     * [CrawlResCommonV1](#CrawlResCommonV1)
     * [CrawlResCommonArrV1](#CrawlResCommonArrV1)
+    * [CrawlPage](#CrawlPage-2) 
     * [FileInfo](#FileInfo)
-    * [CrawlPage](#CrawlPage)
 - [More](#More)
 
 ## Install
@@ -682,10 +681,21 @@ interface AnyObject extends Object {
 type Method = 'get' | 'GET' | 'delete' | 'DELETE' | 'head' | 'HEAD' | 'options' | 'OPTONS' | 'post' | 'POST' | 'put' | 'PUT' | 'patch' | 'PATCH' | 'purge' | 'PURGE' | 'link' | 'LINK' | 'unlink' | 'UNLINK'
 ```
 
-### RequestConfigObject
+### RequestConfigObjectV1
+
+```ts 
+interface RequestConfigObjectV1 {
+  url: string
+  headers?: AnyObject
+  timeout?: number
+  proxy?: string
+}
+```
+
+### RequestConfigObjectV2
 
 ```ts 
-interface RequestConfigObject {
+interface RequestConfigObjectV2 {
   url: string
   method?: Method
   headers?: AnyObject
@@ -699,17 +709,7 @@ interface RequestConfigObject {
 ### RequestConfig
 
 ```ts
-type RequestConfig = string | RequestConfigObject
-```
-
-### MergeRequestConfigObject
-
-```ts
-interface MergeRequestConfigObject {
-  url: string
-  timeout?: number
-  proxy?: string
-}
+type RequestConfig = string | RequestConfigObjectV2
 ```
 
 ### IntervalTime
@@ -733,6 +733,12 @@ interface XCrawlBaseConfig {
 }
 ```
 
+### CrawlPageConfig
+
+```ts
+type CrawlPageConfig = string | RequestConfigObjectV1
+```
+
 ### CrawlBaseConfigV1
 
 ```ts
@@ -742,12 +748,6 @@ interface CrawlBaseConfigV1 {
 }
 ```
 
-### CrawlPageConfig
-
-```ts
-type CrawlPageConfig = string | MergeRequestConfigObject
-```
-
 ### CrawlDataConfig
 
 ```ts
@@ -805,7 +805,7 @@ interface XCrawlInstance {
 ### CrawlResCommonV1
 
 ```ts
-interface CrawlCommon<T> {
+interface CrawlResCommonV1<T> {
   id: number
   statusCode: number | undefined
   headers: IncomingHttpHeaders // nodejs: http type
@@ -819,6 +819,17 @@ interface CrawlCommon<T> {
 type CrawlResCommonArrV1<T> = CrawlResCommonV1<T>[]
 ```
 
+### CrawlPage
+
+```ts
+interface CrawlPage {
+  httpResponse: HTTPResponse | null // The type of HTTPResponse in the puppeteer library
+  browser: Browser // The Browser type of the puppeteer library
+  page: Page // The Page type of the puppeteer library
+  jsdom: JSDOM // jsdom type of the JSDOM library
+}
+```
+
 ### FileInfo
 
 ```ts
@@ -830,17 +841,6 @@ interface FileInfo {
 }
 ```
 
-### CrawlPage
-
-```ts
-interface CrawlPage {
-  httpResponse: HTTPResponse | null // The type of HTTPResponse in the puppeteer library
-  browser // The type of Browser in the puppeteer library
-  page: Page // The type of Page in the puppeteer library
-  jsdom: JSDOM // The type of JSDOM in the jsdom library
-}
-```
-
 ## More
 
 If you have any **questions** or **needs** , please submit **Issues in** https://github.com/coder-hxl/x-crawl/issues .

docs/cn.md

@@ -9,12 +9,12 @@ x-crawl 是一个灵活的 nodejs 爬虫库。可以爬取页面并控制页面
 ## 特征
 
 - 支持 异步/同步 方式爬取数据。
-- 写法非常灵活，支持多种方式写请求配置和获取爬取结果。
+- 灵活的写法，支持多种方式写请求配置和获取爬取结果。
 - 灵活的爬取间隔时间，由你决定 使用/避免 高并发爬取。
 - 简单的配置即可抓取页面、批量网络请求以及批量下载文件资源等操作。
 - 拥有轮询功能，定时爬取数据。
 - 内置 puppeteer 爬取页面，并用采用 jsdom 库对页面内容解析，也支持自行解析。
-- 对批量爬取的成功和失败进行捕获记录，并进行高亮的提醒。
+- 对爬取的成功和失败进行捕获记录，并进行高亮的提醒。
 - 使用 TypeScript 编写，拥有类型，提供泛型。
 
 ## 跟 puppeteer 的关系
@@ -65,20 +65,20 @@ crawlPage API 的返回值将可以做以下操作:
 - [类型](#类型-6)
     * [AnyObject](#AnyObject)
     * [Method](#Method)
-    * [RequestConfigObject](#RequestConfigObject)
+    * [RequestConfigObjectV1](#RequestConfigObjectV1)
+    * [RequestConfigObjectV2](#RequestConfigObjectV2)
     * [RequestConfig](#RequestConfig)
-    * [MergeRequestConfigObject](#MergeRequestConfigObject)
     * [IntervalTime](#IntervalTime)
     * [XCrawlBaseConfig](#XCrawlBaseConfig)
-    * [CrawlBaseConfigV1](#CrawlBaseConfigV1)
     * [CrawlPageConfig](#CrawlPageConfig	)
+    * [CrawlBaseConfigV1](#CrawlBaseConfigV1)
     * [CrawlDataConfig](#CrawlDataConfig) 
     * [CrawlFileConfig](#CrawlFileConfig)
     * [StartPollingConfig](#StartPollingConfig)
     * [CrawlResCommonV1](#CrawlResCommonV1)
     * [CrawlResCommonArrV1](#CrawlResCommonArrV1)
+    * [CrawlPage](#CrawlPage-2) 
     * [FileInfo](#FileInfo)
-    * [CrawlPage](#CrawlPage) 
 - [更多](#更多)
 
 ## 安装
@@ -673,10 +673,21 @@ interface AnyObject extends Object {
 type Method = 'get' | 'GET' | 'delete' | 'DELETE' | 'head' | 'HEAD' | 'options' | 'OPTONS' | 'post' | 'POST' | 'put' | 'PUT' | 'patch' | 'PATCH' | 'purge' | 'PURGE' | 'link' | 'LINK' | 'unlink' | 'UNLINK'
 ```
 
-### RequestConfigObject
+### RequestConfigObjectV1
 
 ```ts 
-interface RequestConfigObject {
+interface RequestConfigObjectV1 {
+  url: string
+  headers?: AnyObject
+  timeout?: number
+  proxy?: string
+}
+```
+
+### RequestConfigObjectV2
+
+```ts 
+interface RequestConfigObjectV2 {
   url: string
   method?: Method
   headers?: AnyObject
@@ -690,17 +701,7 @@ interface RequestConfigObject {
 ### RequestConfig
 
 ```ts
-type RequestConfig = string | RequestConfigObject
-```
-
-### MergeRequestConfigObject
-
-```ts
-interface MergeRequestConfigObject {
-  url: string
-  timeout?: number
-  proxy?: string
-}
+type RequestConfig = string | RequestConfigObjectV2
 ```
 
 ### IntervalTime
@@ -724,6 +725,12 @@ interface XCrawlBaseConfig {
 }
 ```
 
+### CrawlPageConfig
+
+```ts
+type CrawlPageConfig = string | RequestConfigObjectV1
+```
+
 ### CrawlBaseConfigV1
 
 ```ts
@@ -733,12 +740,6 @@ interface CrawlBaseConfigV1 {
 }
 ```
 
-### CrawlPageConfig
-
-```ts
-type CrawlPageConfig = string | MergeRequestConfigObject
-```
-
 ### CrawlDataConfig
 
 ```ts
@@ -810,17 +811,6 @@ interface CrawlResCommonV1<T> {
 type CrawlResCommonArrV1<T> = CrawlResCommonV1<T>[]
 ```
 
-### FileInfo
-
-```ts
-interface FileInfo {
-  fileName: string
-  mimeType: string
-  size: number
-  filePath: string
-}
-```
-
 ### CrawlPage
 
 ```ts
@@ -832,6 +822,17 @@ interface CrawlPage {
 }
 ```
 
+### FileInfo
+
+```ts
+interface FileInfo {
+  fileName: string
+  mimeType: string
+  size: number
+  filePath: string
+}
+```
+
 ## 更多
 
 如有 **问题** 或 **需求** 请在 https://github.com/coder-hxl/x-crawl/issues 中提 **Issues** 。

package.json

@@ -1,7 +1,7 @@
 {
   "private": true,
   "name": "x-crawl",
-  "version": "3.2.7",
+  "version": "3.2.8",
   "author": "coderHXL",
   "description": "x-crawl is a flexible nodejs crawler library.",
   "license": "MIT",
@@ -39,6 +39,6 @@
     "rollup": "^3.10.1",
     "rollup-plugin-typescript2": "^0.34.1",
     "ts-jest": "^29.0.5",
-    "typescript": "^4.9.4"
+    "typescript": "5.0.2"
   }
 }