Certain websites with HTTP/2 only protocol implementations return StreamReset stream_id:1 [Detailed Repro]

[0xCognition]

Issue Description:

I encountered an issue when making simple GET requests to certain websites that implement only the HTTP/2 protocol (that do not have a fallback to HTTP/1.1). The requests result in a stream reset error.

Interestingly, this issue is resolved when the requests are proxied through tools like mitmproxy (specifically mitmweb). However, using mitmproxy as an upstream proxy isn't a desired end state.

Steps to Reproduce:

1. Use httpx to make a simple GET request to https://www.homes.com/ (PoC below)
2. Observe that a stream reset error occurs:
   httpx.RemoteProtocolError: <StreamReset stream_id:1, error_code:ErrorCodes.INTERNAL_ERROR, remote_reset:True>

Expected Behavior:

The GET request should succeed without requiring additional proxying.

Observed Behavior:

The GET request fails with a stream reset error.

Workaround:

Proxied requests through mitmproxy do not exhibit the issue.

Environment:

Tested httpx versions: 0.27.2, 0.28.1
Tested Python version: 3.10
Tested Operating Systems: Ubuntu 22.04

---

Python PoC

import httpx

url = 'https://www.homes.com/'

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7',
'Sec-Fetch-Mode': 'navigate',
'Sec-Fetch-User': '?1',
'Sec-Fetch-Dest': 'document',
'Referer': 'https://www.homes.com'
}

client = httpx.Client(http2=True, verify=False)
response = client.get(url, headers=headers)
print(response.status_code)

---

Stack Trace

Traceback (most recent call last):
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 72, in map_httpcore_exceptions
yield
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 236, in handle_request
resp = self._pool.handle_request(req)
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
raise exc
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
response = connection.handle_request(request)
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/connection.py", line 103, in handle_request
return self._connection.handle_request(request)
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/http2.py", line 185, in handle_request
raise exc
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/http2.py", line 148, in handle_request
status, headers = self._receive_response(
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/http2.py", line 292, in _receive_response
event = self._receive_stream_event(request, stream_id)
File "/home/script/.venv/lib/python3.10/site-packages/httpcore/_sync/http2.py", line 336, in _receive_stream_event
raise RemoteProtocolError(event)
httpcore.RemoteProtocolError: <StreamReset stream_id:1, error_code:ErrorCodes.INTERNAL_ERROR, remote_reset:True>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/script/test/http2_tester.py", line 43, in
response = client.get(url, headers=headers)
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1066, in get
return self.request(
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 837, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 926, in send
response = self._send_handling_auth(
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 954, in _send_handling_auth
response = self._send_handling_redirects(
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 991, in _send_handling_redirects
response = self._send_single_request(request)
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_client.py", line 1027, in _send_single_request
response = transport.handle_request(request)
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 235, in handle_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in exit
self.gen.throw(typ, value, traceback)
File "/home/script/.venv/lib/python3.10/site-packages/httpx/_transports/default.py", line 89, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.RemoteProtocolError: <StreamReset stream_id:1, error_code:ErrorCodes.INTERNAL_ERROR, remote_reset:True>

[tomchristie]

cross check against curl confirms this as a server issue...

curl https://www.homes.com
curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

[0xCognition]

By the way, trying the following does yield a 400 response from the server indicating a likely intentional configuration on their part to prevent things like curl, httpx, etc. from working. The key here is though, curl does not encounter the reset error when using the custom user agent, even though httpx does.

curl https://www.homes.com/ -A "Mozilla"

<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
 
You don't have permission to access "http&#58;&#47;&#47;www&#46;homes&#46;com&#47;" on this server.<P>
Reference&#32;&#35;18&#46;45593a17&#46;1734981077&#46;13eaa6ae
<P>https&#58;&#47;&#47;errors&#46;edgesuite&#46;net&#47;18&#46;45593a17&#46;1734981077&#46;13eaa6ae</P>
</BODY>
</HTML>

[0xCognition]

Hi Tom --- Thanks for the quick reply. This is either some type of accidental server misconfiguration as you pointed out, or possibly an intentional misconfiguration to prevent HTTP clients from connecting. That said, my thought was --- from a user's perspective, they might simply expect their HTTP client to "just work," even when dealing with potentially non-compliant servers. This site happens to not implement a fallback to HTTP/1.1, so conceivably non-compliant servers could be a bigger problem in the future --- especially if intentional.

That said, if normal protocol is to not attempt to fix these type of issues --- I totally understand. Appreciate your work!

[adrichman]

For what it's worth @0xCognition, with this particular example, the request failed on both macOS and linux platforms. I was able to get it to succeed on macOS by including headers that are also sent by the web browser e.g.

    "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "accept-encoding": "gzip, deflate, br",
    "accept-language": "en-US,en;q=0.9",
    "pragma": "no-cache",
    "priority": "u=0, i",
    "sec-ch-ua": '"Google Chrome";v="131", "Chromium";v="131", "Not_A Brand";v="24"',
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "macOS",
    "sec-fetch-dest": "document",
    "sec-fetch-mode": "cors",
    "sec-fetch-site": "cross-site",
    "sec-fetch-user": "?1",
    "upgrade-insecure-requests": "1",
    "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",

Additionally, I needed to ensure the sec-ch-ua-platform header is accurate (e.g. I need to pass "sec-ch-ua-platform": "Unknown" when running this on ubuntu in order to get a valid response – however, it's a 403)

Lastly, I needed to use an OrderedDict when providing all the headers, ensuring that the headers were provided in the same order that the server expected from the client.

[0xCognition]

Thanks for the additional info, Adam. I'll need to give the OrderedDict approach a try.