Add cli support (#1855)

* Add cli support

* Add setup.py

* Import main to 'httpx.main'

* Add 'cli' to requirements

* Add tests for command-line client

* Drop most CLI tests

* Add test_json

* Add test_redirects

* Coverage exclusion over _main.py in order to test more clearly

* Black formatting

* Add test_follow_redirects

* Add test_post, test_verbose, test_auth

* Add test_errors

* Remove test_errors

* Add test_download

* Change test_errors - perhaps the empty host header was causing the socket error?

* Update test_errors to not break socket

* Update docs

* Update version to 1.0.0.beta0

* Tweak CHANGELOG

* Fix up images in README

* Tweak images in README

* Update README

Author: lovelydinosaur

CHANGELOG.md

@@ -4,6 +4,73 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## 1.0.0.beta0
+
+The 1.0 pre-release adds an integrated command-line client, and also includes some
+design changes. The most notable of these is that redirect responses are no longer
+automatically followed, unless specifically requested.
+
+This design decision prioritises a more explicit approach to redirects, in order
+to avoid code that unintentionally issues multiple requests as a result of
+misconfigured URLs.
+
+For example, previously a client configured to send requests to `http://api.github.com/`
+would end up sending every API request twice, as each request would be redirected to `https://api.github.com/`.
+
+If you do want auto-redirect behaviour, you can enable this either by configuring
+the client instance with `Client(follow_redirects=True)`, or on a per-request
+basis, with `.get(..., follow_redirects=True)`.
+
+This change is a classic trade-off between convenience and precision, with no "right"
+answer. See [discussion #1785](https://github.com/encode/httpx/discussions/1785) for more
+context.
+
+The other major design change is an update to the Transport API, which is the low-level
+interface against which requests are sent. Previously this interface used only primitive
+datastructures, like so...
+
+```python
+(status_code, headers, stream, extensions) = transport.handle_request(method, url, headers, stream, extensions)
+try
+    ...
+finally:
+    stream.close()
+```
+
+Now the interface is much simpler...
+
+```python
+response = transport.handle_request(request)
+try
+    ...
+finally:
+    response.close()
+```
+
+### Changed
+
+* The `allow_redirects` flag is now `follow_redirects` and defaults to `False`.
+* The `raise_for_status()` method will now raise an exception for any responses
+  except those with 2xx status codes. Previously only 4xx and 5xx status codes
+  would result in an exception.
+* The low-level transport API changes to the much simpler `response = transport.handle_request(request)`.
+* The `client.send()` method no longer accepts a `timeout=...` argument, but the
+  `client.build_request()` does. This required by the signature change of the
+  Transport API. The request timeout configuration is now stored on the request
+  instance, as `request.extensions['timeout']`.
+
+### Added
+
+* Added the `httpx` command-line client.
+* Response instances now include `.is_informational`, `.is_success`, `.is_redirect`, `.is_client_error`, and `.is_server_error`
+  properties for checking 1xx, 2xx, 3xx, 4xx, and 5xx response types. Note that the behaviour of `.is_redirect` is slightly different in that it now returns True for all 3xx responses, in order to allow for a consistent set of properties onto the different HTTP status code types. The `response.has_redirect_location` location may be used to determine responses with properly formed URL redirects.
+
+### Fixed
+
+* `response.iter_bytes()` no longer raises a ValueError when called on a response with no content. (Pull #1827)
+* The `'wsgi.error'` configuration now defaults to `sys.stderr`, and is corrected to be a `TextIO` interface, not a `BytesIO` interface. Additionally, the WSGITransport now accepts a `wsgi_error` confguration. (Pull #1828)
+* Follow the WSGI spec by properly closing the iterable returned by the application. (Pull #1830)
+
 ## 0.19.0 (19th August, 2021)
 
 ### Added

README.md

@@ -13,15 +13,21 @@
 </a>
 </p>
 
-HTTPX is a fully featured HTTP client for Python 3, which provides sync and async APIs, and support for both HTTP/1.1 and HTTP/2.
+HTTPX is a fully featured HTTP client library for Python 3. It includes **an integrated
+command line client**, has support for both **HTTP/1.1 and HTTP/2**, and provides both **sync
+and async APIs**.
 
-**Note**: _HTTPX should be considered in beta. We believe we've got the public API to
-a stable point now, but would strongly recommend pinning your dependencies to the `0.19.*`
-release, so that you're able to properly review [API changes between package updates](https://github.com/encode/httpx/blob/master/CHANGELOG.md). A 1.0 release is expected to be issued sometime in 2021._
+**Note**: *This is the README for the 1.0 pre-release. This release adds support for an integrated command-line client, and also includes a couple of design changes from 0.19. Redirects are no longer followed by default, and the low-level Transport API has been updated. Upgrades from 0.19 will need to see [the CHANGELOG](https://github.com/encode/httpx/blob/version-1.0/CHANGELOG.md) for more details.*
 
 ---
 
-Let's get started...
+Installing HTTPX.
+
+```shell
+$ pip install httpx --pre
+```
+
+Now, let's get started...
 
 ```pycon
 >>> import httpx
@@ -36,26 +42,32 @@ Let's get started...
 '<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'
 ```
 
-Or, using the async API...
-
-_Use [IPython](https://ipython.readthedocs.io/en/stable/) or Python 3.8+ with `python -m asyncio` to try this code interactively._
+Or, using the command-line client.
 
-```pycon
->>> import httpx
->>> async with httpx.AsyncClient() as client:
-...     r = await client.get('https://www.example.org/')
-...
->>> r
-<Response [200 OK]>
+```shell
+$ pip install --pre 'httpx[cli]'  # The command line client is an optional dependency.
 ```
 
+Which now allows us to use HTTPX directly from the command-line...
+
+<p align="center">
+  <img width="700" src="docs/img/httpx-help.png" alt='httpx --help'>
+</p>
+
+Sending a request...
+
+<p align="center">
+  <img width="700" src="docs/img/httpx-request.png" alt='httpx http://httpbin.org/json'>
+</p>
+
 ## Features
 
 HTTPX builds on the well-established usability of `requests`, and gives you:
 
 * A broadly [requests-compatible API](https://www.python-httpx.org/compatibility/).
-* Standard synchronous interface, but with [async support if you need it](https://www.python-httpx.org/async/).
+* An integrated command-line client.
 * HTTP/1.1 [and HTTP/2 support](https://www.python-httpx.org/http2/).
+* Standard synchronous interface, but with [async support if you need it](https://www.python-httpx.org/async/).
 * Ability to make requests directly to [WSGI applications](https://www.python-httpx.org/advanced/#calling-into-python-web-apps) or [ASGI applications](https://www.python-httpx.org/async/#calling-into-python-web-apps).
 * Strict timeouts everywhere.
 * Fully type annotated.

docs/compatibility.md

@@ -1,29 +1,10 @@
 # Requests Compatibility Guide
 
-HTTPX aims to be broadly compatible with the `requests` API.
+HTTPX aims to be broadly compatible with the `requests` API, although there are a
+few design differences in places.
 
 This documentation outlines places where the API differs...
 
-## Client instances
-
-The HTTPX equivalent of `requests.Session` is `httpx.Client`.
-
-```python
-session = requests.Session(**kwargs)
-```
-
-is generally equivalent to
-
-```python
-client = httpx.Client(**kwargs)
-```
-
-## Request URLs
-
-Accessing `response.url` will return a `URL` instance, rather than a string.
-
-Use `str(response.url)` if you need a string instance.
-
 ## Redirects
 
 Unlike `requests`, HTTPX does **not follow redirects by default**.
@@ -44,6 +25,26 @@ Or else instantiate a client, with redirect following enabled by default...
 client = httpx.Client(follow_redirects=True)
 ```
 
+## Client instances
+
+The HTTPX equivalent of `requests.Session` is `httpx.Client`.
+
+```python
+session = requests.Session(**kwargs)
+```
+
+is generally equivalent to
+
+```python
+client = httpx.Client(**kwargs)
+```
+
+## Request URLs
+
+Accessing `response.url` will return a `URL` instance, rather than a string.
+
+Use `str(response.url)` if you need a string instance.
+
 ## Determining the next redirect request
 
 The `requests` library exposes an attribute `response.next`, which can be used to obtain the next redirect request.
@@ -97,8 +98,7 @@ opened in text mode.
 ## Content encoding
 
 HTTPX uses `utf-8` for encoding `str` request bodies. For example, when using `content=<str>` the request body will be encoded to `utf-8` before being sent over the wire. This differs from Requests which uses `latin1`. If you need an explicit encoding, pass encoded bytes explictly, e.g. `content=<str>.encode("latin1")`.
-
-For response bodies, assuming the server didn't send an explicit encoding then HTTPX will do its best to figure out an appropriate encoding. HTTPX makes a guess at the encoding to use for decoding the response using `charset_normalizer`. Fallback to that or any content with less than 32 octets will be decoded using `utf-8` with the `error="replace"` decoder strategy.  
+For response bodies, assuming the server didn't send an explicit encoding then HTTPX will do its best to figure out an appropriate encoding. HTTPX makes a guess at the encoding to use for decoding the response using `charset_normalizer`. Fallback to that or any content with less than 32 octets will be decoded using `utf-8` with the `error="replace"` decoder strategy.
 
 ## Cookies
 
@@ -133,7 +133,7 @@ HTTPX provides a `.stream()` interface rather than using `stream=True`. This ens
 For example:
 
 ```python
-with request.stream("GET", "https://www.example.com") as response:
+with httpx.stream("GET", "https://www.example.com") as response:
     ...
 ```
 
@@ -165,13 +165,21 @@ Requests supports `REQUESTS_CA_BUNDLE` which points to either a file or a direct
 
 ## Request body on HTTP methods
 
-The HTTP `GET`, `DELETE`, `HEAD`, and `OPTIONS` methods are specified as not supporting a request body. To stay in line with this, the `.get`, `.delete`, `.head` and `.options` functions do not support `files`, `data`, or `json` arguments.
+The HTTP `GET`, `DELETE`, `HEAD`, and `OPTIONS` methods are specified as not supporting a request body. To stay in line with this, the `.get`, `.delete`, `.head` and `.options` functions do not support `content`, `files`, `data`, or `json` arguments.
 
 If you really do need to send request data using these http methods you should use the generic `.request` function instead.
 
-## Checking for 4xx/5xx responses
+```python
+httpx.request(
+  method="DELETE",
+  url="https://www.example.com/",
+  content=b'A request body on a DELETE request.'
+)
+```
+
+## Checking for success and failure responses
 
-We don't support `response.is_ok` since the naming is ambiguous there, and might incorrectly imply an equivalence to `response.status_code == codes.OK`. Instead we provide the `response.is_error` property. Use `if not response.is_error:` instead of `if response.is_ok:`.
+We don't support `response.is_ok` since the naming is ambiguous there, and might incorrectly imply an equivalence to `response.status_code == codes.OK`. Instead we provide the `response.is_success` property, which can be used to check for a 2xx response.
 
 ## Request instantiation
 

docs/img/httpx-help.png

@@ -25,15 +25,19 @@ HTTPX is a fully featured HTTP client for Python 3, which provides sync and asyn
 
 
 !!! note
-    HTTPX should currently be considered in beta.
+    This is the documentation for the 1.0 pre-release.
 
-    We believe we've got the public API to a stable point now, but would strongly recommend pinning your dependencies to the `0.19.*` release, so that you're able to properly review [API changes between package updates](https://github.com/encode/httpx/blob/master/CHANGELOG.md).
-
-    A 1.0 release is expected to be issued sometime in 2021.
+    This release adds support for an integrated command-line client, and also includes a couple of design changes from 0.19. Redirects are no longer followed by default, and the low-level Transport API has been updated. See [the CHANGELOG](https://github.com/encode/httpx/blob/version-1.0/CHANGELOG.md) for more details.
 
 ---
 
-Let's get started...
+Installing the HTTPX 1.0 pre-release.
+
+```shell
+$ pip install httpx --pre
+```
+
+Now, let's get started...
 
 ```pycon
 >>> import httpx
@@ -48,23 +52,24 @@ Let's get started...
 '<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'
 ```
 
-Or, using the async API...
-
-_Use [IPython](https://ipython.readthedocs.io/en/stable/) or Python 3.8+ with `python -m asyncio` to try this code interactively._
+Or, using the command-line client.
 
-```pycon
->>> import httpx
->>> async with httpx.AsyncClient() as client:
-...     r = await client.get('https://www.example.org/')
-...
->>> r
-<Response [200 OK]>
+```shell
+# The command line client is an optional dependency.
+$ pip install --pre 'httpx[cli]'
 ```
 
+Which now allows us to use HTTPX directly from the command-line...
+
+![httpx --help](img/httpx-help.png)
+
+Sending a request...
+
+![httpx http://httpbin.org/json](img/httpx-request.png)
+
 ## Features
 
-HTTPX is a high performance asynchronous HTTP client, that builds on the
-well-established usability of `requests`, and gives you:
+HTTPX builds on the well-established usability of `requests`, and gives you:
 
 * A broadly [requests-compatible API](compatibility.md).
 * Standard synchronous interface, but with [async support if you need it](async.md).

docs/img/httpx-request.png

@@ -73,9 +73,7 @@ You can inspect what encoding will be used to decode the response.
 ```
 
 In some cases the response may not contain an explicit encoding, in which case HTTPX
-will attempt to automatically determine an encoding to use. This defaults to
-UTF-8, but also includes robust fallback behaviour for handling ascii,
-iso-8859-1 and windows 1252 encodings.
+will attempt to automatically determine an encoding to use.
 
 ```pycon
 >>> r.encoding
@@ -84,7 +82,6 @@ None
 '<!doctype html>\n<html>\n<head>\n<title>Example Domain</title>...'
 ```
 
-
 If you need to override the standard behaviour and explicitly set the encoding to
 use, then you can do that too.
 
@@ -277,7 +274,7 @@ HTTPX also includes an easy shortcut for accessing status codes by their text ph
 True
 ```
 
-We can raise an exception for any Client or Server error responses (4xx or 5xx status codes):
+We can raise an exception for any responses which are not a 2xx success code:
 
 ```pycon
 >>> not_found = httpx.get('https://httpbin.org/status/404')

docs/index.md

@@ -43,6 +43,21 @@
 from ._transports.wsgi import WSGITransport
 from ._types import AsyncByteStream, SyncByteStream
 
+try:
+    from ._main import main
+except ImportError:  # pragma: nocover
+
+    def main() -> None:  # type: ignore
+        import sys
+
+        print(
+            "The httpx command line client could not run because the required "
+            "dependencies were not installed.\nMake sure you've installed "
+            "everything with: pip install 'httpx[cli]'"
+        )
+        sys.exit(1)
+
+
 __all__ = [
     "__description__",
     "__title__",
@@ -76,6 +91,7 @@
     "InvalidURL",
     "Limits",
     "LocalProtocolError",
+    "main",
     "MockTransport",
     "NetworkError",
     "options",

docs/quickstart.md

@@ -1,3 +1,3 @@
 __title__ = "httpx"
 __description__ = "A next generation HTTP client, for Python 3."
-__version__ = "0.19.0"
+__version__ = "1.0.0.beta0"