Skip to content

Commit 1e8d49b

Browse files
lexiforestlexiforest
committed
Update docs
1 parent f3026a5 commit 1e8d49b

6 files changed

Lines changed: 82 additions & 243 deletions

File tree

‎INSTALL.md‎

Lines changed: 1 addition & 161 deletions
Original file line numberDiff line numberDiff line change
@@ -1,161 +1 @@
1-
# Building and installing curl-impersonate
2-
3-
This guide shows how to compile and install curl-impersonate and libcurl-impersonate from source.
4-
The build process takes care of downloading dependencies, patching them, compiling them and finally compiling curl itself with the needed patches.
5-
There are currently three build options depending on your use case:
6-
7-
* [Native build](#native-build) using an autotools-based Makefile
8-
* [Cross compiling](#cross-compiling) using an autotools-based Makefile
9-
* [Docker container build](#docker-build)
10-
11-
Unlike the upstream project, there is only one version in this fork, namely the Chrome version, for impersonating all main stream browsers.
12-
13-
## Native build
14-
15-
### Ubuntu
16-
17-
Install dependencies for building all the components:
18-
19-
```sh
20-
sudo apt install build-essential pkg-config cmake ninja-build curl autoconf automake libtool
21-
sudo apt install golang-go unzip
22-
sudo apt install zstd libzstd-dev
23-
```
24-
25-
Clone this repository:
26-
27-
```sh
28-
git clone https://github.com/lexiforest/curl-impersonate.git
29-
cd curl-impersonate
30-
```
31-
32-
Configure and compile:
33-
34-
```sh
35-
mkdir build && cd build
36-
../configure
37-
# Build and install
38-
make build
39-
sudo make install
40-
# You may need to update the linker's cache to find libcurl-impersonate
41-
sudo ldconfig
42-
# Optionally remove all the build files
43-
cd ../ && rm -Rf build
44-
```
45-
46-
This will install curl-impersonate, libcurl-impersonate and the wrapper scripts to `/usr/local`. To change the installation path, pass `--prefix=/path/to/install/` to the `configure` script.
47-
48-
After installation you can run the wrapper scripts, e.g.:
49-
50-
```sh
51-
curl_chrome119 https://www.example.com
52-
```
53-
54-
or run directly with you own flags:
55-
56-
```sh
57-
curl-impersonate https://www.example.com
58-
```
59-
60-
### Red Hat based (CentOS/Fedora/Amazon Linux)
61-
62-
Install dependencies:
63-
64-
```sh
65-
yum groupinstall "Development Tools"
66-
yum groupinstall "C Development Tools and Libraries" # Fedora only
67-
yum install cmake3 python3 python3-pip
68-
# Install Ninja. This may depend on your system.
69-
yum install ninja-build
70-
# OR
71-
pip3 install ninja
72-
yum install zstd libzstd-devel
73-
```
74-
75-
For the Chrome version, install Go.
76-
You may need to follow the [Go installation instructions](https://go.dev/doc/install) if it's not packaged for your system:
77-
78-
```sh
79-
yum install golang
80-
```
81-
82-
Then follow the 'Ubuntu' instructions for the actual build.
83-
84-
### macOS
85-
86-
Install dependencies for building all the components:
87-
88-
```sh
89-
brew install pkg-config make cmake ninja autoconf automake libtool
90-
brew install zstd
91-
brew install go
92-
```
93-
94-
Clone this repository:
95-
96-
```sh
97-
git clone https://github.com/lexiforest/curl-impersonate.git
98-
cd curl-impersonate
99-
```
100-
101-
Configure and compile:
102-
103-
```sh
104-
mkdir build && cd build
105-
../configure
106-
# Build and install
107-
gmake build
108-
sudo gmake install
109-
# Optionally remove all the build files
110-
cd ../ && rm -Rf build
111-
```
112-
113-
### Static compilation
114-
115-
To compile curl-impersonate statically with libcurl-impersonate, pass `--enable-static` to the `configure` script.
116-
117-
118-
## Cross compiling
119-
120-
There is some basic support for cross compiling curl-impersonate.
121-
It is currently being used to build curl-impersonate for ARM64 (aarch64) systems from x86-64 systems.
122-
Cross compiling is similar to the usual build but a bit trickier:
123-
124-
* You'd have to build zlib and zstd for the target architecture so that curl can link with it.
125-
* Some paths have to be specified manually since curl's own build system can't determine their location.
126-
127-
An example build for aarch64 on Ubuntu x86_64:
128-
129-
```sh
130-
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
131-
132-
./configure --host=aarch64-linux-gnu \
133-
--with-zlib=/path/to/compiled/zlib \
134-
--with-zstd=/path/to/compiled/zstd \
135-
--with-ca-path=/etc/ssl/certs \
136-
--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt
137-
138-
make build
139-
```
140-
141-
The flags mean as follows:
142-
`--with-zlib/zstd` is the location of a compiled zlib/zstd library for the target architecture.
143-
`--with-ca-path` and `--with-ca-bundle` will be passed to curl's configure script as is.
144-
145-
## Docker build
146-
147-
The Docker build is a bit more reproducible and serves as the reference implementation. It creates a Debian-based Docker image with the binaries.
148-
149-
[`chrome/Dockerfile`](chrome/Dockerfile) is a debian-based Dockerfile that will build curl with all the necessary modifications and patches. Build it like the following:
150-
151-
```sh
152-
docker build -t curl-impersonate chrome/
153-
```
154-
155-
The resulting binaries and libraries are in the `/usr/local` directory, which contains:
156-
157-
* `curl-impersonate`, - The curl binary that can impersonate Chrome/Edge/Safari. It is compiled statically against libcurl, BoringSSL, and libnghttp2 so that it won't conflict with any existing libraries on your system. You can use it from the container or copy it out. Tested to work on Ubuntu 20.04.
158-
* `curl_chrome99`, `curl_chrome100`, `...` - Wrapper scripts that launch `curl-impersonate` with all the needed flags.
159-
* `libcurl-impersonate.so`, `libcurl-impersonate.so` - libcurl compiled with impersonation support. See [libcurl-impersonate](README.md#libcurl-impersonate) for more details.
160-
161-
You can use them inside the docker, copy them out using `docker cp` or use them in a multi-stage docker build.
1+
See docs/install.rst and docs/building.rst.

‎docs/api.rst‎

Lines changed: 24 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,10 @@ The following ``CURLOPT_*`` options and CLI arguments are added by curl-imperson
55
are not part of upstream curl.
66

77
Many of them are applied automatically by ``curl_easy_impersonate()`` given a preset
8-
fingerprint. In particular,
9-
the impersonation helper sets browser-like TLS, HTTP/2, and HTTP/3 options such as
10-
``CURLOPT_HTTPBASEHEADER``, ``CURLOPT_HTTP2_PSEUDO_HEADERS_ORDER``,
11-
``CURLOPT_HTTP2_SETTINGS``, ``CURLOPT_HTTP2_WINDOW_UPDATE``,
12-
``CURLOPT_SSL_ENABLE_ALPS``, ``CURLOPT_SSL_SIG_HASH_ALGS``,
13-
``CURLOPT_SSL_CERT_COMPRESSION``, ``CURLOPT_SSL_ENABLE_TICKET``,
14-
``CURLOPT_TLS_GREASE``, and ``CURLOPT_TLS_EXTENSION_ORDER``.
8+
fingerprint.
9+
10+
If you want to customize the fingerprint, or create your own impersonation target, below
11+
is all the options we have added on top of vanilla curl:
1512

1613
Impersonation and Headers
1714
-------------------------
@@ -29,38 +26,42 @@ Impersonation and Headers
2926
Command line: no direct equivalent.
3027

3128
``CURLOPT_HTTPHEADER_ORDER`` (string)
32-
Comma-separated order for normal HTTP headers.
29+
Comma-separated order for normal HTTP headers. e.g. ``Host,User-Agent,Cookie``. This
30+
is particularly useful for impersonating the http/1.1 behavior.
3331
Command line: ``--http-header-order <headers>``.
3432

3533
``CURLOPT_FORM_BOUNDARY`` (string)
36-
Sets the multipart ``form-data`` boundary style.
34+
Sets the multipart ``form-data`` boundary style. Possible values: ``webkit``, for
35+
webkit and blink based browsers, e.g. Safari and Chrome. ``firefox``, for Gecko based
36+
browsers, e.g. Firefox.
3737
Command line: no direct equivalent.
3838

3939
TLS
4040
---
4141

4242
``CURLOPT_SSL_SIG_HASH_ALGS`` (string)
43-
Sets the TLS signature hash algorithms. The patch notes that upstream curl later
43+
Sets the TLS signature hash algorithms. Note that upstream curl later
4444
implemented a similar option as option 328, but curl-impersonate keeps this name for
4545
compatibility. See RFC 5246 section 7.4.1.4.1.
4646
Command line: ``--signature-hashes <algorithm list>``.
4747

4848
``CURLOPT_SSL_ENABLE_ALPS`` (long)
49-
Enables or disables ALPS in TLS. The patch notes that ALPS support here is minimal and
50-
is intended only to make the TLS ClientHello match browser behavior.
49+
Enables or disables ALPS in TLS. Note that recent versions of Chrome started using a
50+
new ID for ALPS.
5151
Command line: ``--alps``.
5252

5353
``CURLOPT_SSL_CERT_COMPRESSION`` (string)
5454
Comma-separated list of certificate compression algorithms to advertise in the TLS
55-
ClientHello. Supported values are ``zlib`` and ``brotli``. See RFC 8879.
55+
ClientHello. Supported values are ``zlib`` and ``brotli`` and ``zstd``. See RFC 8879.
5656
Command line: ``--cert-compression <algorithm list>``.
5757

5858
``CURLOPT_SSL_ENABLE_TICKET`` (long)
5959
Enables or disables the TLS session ticket extension defined by RFC 5077.
6060
Command line: ``--tls-session-ticket`` / ``--no-tls-session-ticket``.
6161

6262
``CURLOPT_SSL_PERMUTE_EXTENSIONS`` (long)
63-
Enables or disables BoringSSL's permuted-extension behavior.
63+
Enables or disables BoringSSL's permuted-extension behavior. This is the default
64+
behavior of Chrome 110 and later.
6465
Command line: ``--tls-permute-extensions``.
6566

6667
``CURLOPT_TLS_GREASE`` (long)
@@ -72,7 +73,7 @@ TLS
7273
Command line: ``--tls-extension-order <order>``.
7374

7475
``CURLOPT_TLS_KEY_USAGE_NO_CHECK`` (long)
75-
Controls the TLS key usage check. The patch comment notes that the default is on.
76+
Disable the TLS key usage check.
7677
Command line: no direct equivalent.
7778

7879
``CURLOPT_TLS_SIGNED_CERT_TIMESTAMPS`` (long)
@@ -84,19 +85,19 @@ TLS
8485
Command line: no direct equivalent.
8586

8687
``CURLOPT_TLS_DELEGATED_CREDENTIALS`` (string)
87-
Controls Firefox-style delegated credentials.
88+
Controls Firefox-style delegated credentials. e.g. ``ecdsa_secp256r1_sha256:ecdsa_secp384r1_sha384:ecdsa_secp521r1_sha512:ecdsa_sha1``
8889
Command line: ``--tls-delegated-credentials <value>``.
8990

9091
``CURLOPT_TLS_RECORD_SIZE_LIMIT`` (long)
91-
Controls Firefox-style TLS record size limit behavior.
92+
Controls Firefox-style TLS record size limit behavior. The typical value is ``4001``
9293
Command line: ``--tls-record-size-limit <integer>``.
9394

9495
``CURLOPT_TLS_KEY_SHARES_LIMIT`` (long)
95-
Controls Firefox-style ``key_shares_limit`` behavior.
96+
Controls Firefox-style ``key_shares_limit`` behavior. The typical value is ``3``
9697
Command line: ``--tls-key-shares-limit <integer>``.
9798

9899
``CURLOPT_TLS_USE_NEW_ALPS_CODEPOINT`` (long)
99-
Uses the new ALPS code point.
100+
Uses the new ALPS codepoint.
100101
Command line: ``--tls-use-new-alps-codepoint``.
101102

102103
HTTP/2
@@ -127,7 +128,7 @@ HTTP/2
127128
Command line: ``--http2-no-priority``.
128129

129130
``CURLOPT_STREAM_EXCLUSIVE`` (long)
130-
Sets HTTP/2 stream exclusiveness. The patch comment documents this as ``0`` or ``1``.
131+
Sets HTTP/2 stream exclusiveness as ``0`` or ``1``.
131132
Command line: ``--http2-stream-exclusive <0|1>``.
132133

133134
HTTP/3 and QUIC
@@ -166,5 +167,7 @@ Proxy and Cookies
166167
Command line: ``--proxy-credential-no-reuse``.
167168

168169
``CURLOPT_SPLIT_COOKIES`` (long)
169-
Splits cookies into separate ``Cookie:`` headers.
170+
Splits cookies into separate ``Cookie:`` headers. For http/1.1, Cookies are joined as
171+
a single header. For http/2 and http/3, Cookies are separated for better compression
172+
rate.
170173
Command line: ``--split-cookies`` / ``--no-split-cookies``.

‎docs/dev.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,3 +4,5 @@ Development and contributing
44
When opening a pull request, do not use the ``main`` branch in your fork. Use a separate
55
branch such as ``feature/xxx`` or ``bugfix/xxx`` so follow-up changes and tests can be
66
added more easily.
7+
8+
See also the AI policy in curl_cffi: https://github.com/lexiforest/curl_cffi#contributing

0 commit comments

Comments
 (0)