Skip to content

Commit 3280699

Browse files
committed
feat: make impit the default HTTP client
1 parent f1f0959 commit 3280699

4 files changed

Lines changed: 9 additions & 9 deletions

File tree

docs/guides/custom-http-client/custom-http-client.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,8 @@ The <ApiLink to="basic-crawler/class/BasicCrawler">`BasicCrawler`</ApiLink> clas
1616

1717
Crawlee provides several HTTP client implementations out of the box:
1818

19-
- **`GotScrapingHttpClient`** (default) - Uses the `got-scraping` library for browser-like requests with support for custom headers, browser fingerprints, and proxies.
20-
- **`ImpitHttpClient`** - Uses the `impit` library for making requests that closely mimic browser behavior.
19+
- **`ImpitHttpClient`** (default) - Uses the `impit` library for making requests that closely mimic browser behavior.
20+
- **`GotScrapingHttpClient`** - Uses the `got-scraping` library for browser-like requests with support for custom headers, browser fingerprints, and proxies. This was the default HTTP client in Crawlee v3.
2121
- **`FetchHttpClient`** - Simple implementation using the native `fetch` API (does not support proxies).
2222

2323
## Implementing a custom HTTP client

packages/basic-crawler/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242
"@apify/timeout": "^0.3.2",
4343
"@apify/utilities": "^2.15.5",
4444
"@crawlee/core": "workspace:*",
45-
"@crawlee/got-scraping-client": "workspace:*",
45+
"@crawlee/impit-client": "workspace:^",
4646
"@crawlee/types": "workspace:*",
4747
"@crawlee/utils": "workspace:*",
4848
"csv-stringify": "^6.5.2",

packages/basic-crawler/src/internals/basic-crawler.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ import {
6161
Statistics,
6262
validators,
6363
} from '@crawlee/core';
64-
import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
64+
import { ImpitHttpClient } from '@crawlee/impit-client';
6565
import type {
6666
Awaitable,
6767
BaseHttpClient,
@@ -377,7 +377,7 @@ export interface BasicCrawlerOptions<
377377

378378
/**
379379
* HTTP client implementation for the `sendRequest` context helper and for plain HTTP crawling.
380-
* Defaults to a new instance of {@apilink GotScrapingHttpClient}
380+
* Defaults to a new instance of {@apilink ImpitHttpClient}
381381
*/
382382
httpClient?: BaseHttpClient;
383383

@@ -807,7 +807,7 @@ export class BasicCrawler<
807807
this.requestManager = new RequestManagerTandem(requestList, () => this.openOwnedRequestQueue());
808808
}
809809

810-
this.httpClient = httpClient ?? new GotScrapingHttpClient({ logger: this.log });
810+
this.httpClient = httpClient ?? new ImpitHttpClient({ logger: this.log });
811811
this.proxyConfiguration = proxyConfiguration;
812812
this.statusMessageLoggingInterval = statusMessageLoggingInterval;
813813
this.statusMessageCallback = statusMessageCallback as StatusMessageCallback;

pnpm-lock.yaml

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)