Header Order / HTTP2 Pseudo Headers

[msanft]

Hey there!

Many of my friends who develop web-scrapers with httpx and I would love if httpx would be a little bit more lax about headers.
Today's sites fingerprint you by so many factors, including the above mentioned header order modifying and the possibilty to custom set HTTP2 pseudo headers. Would be great if we could see options for this in future versions!

[tomchristie]

Help me out a little here by being more specific... 😀

including the above mentioned header order modifying

Could you give a specific example here? What do we currently do wrt. header ordering? What would you like to be able to do?

and the possibilty to custom set HTTP2 pseudo headers.

For example?

[msanft]

Sorry for being unclear! :)

The header order seems to be modified by httpx. I have an ordered dict for my headers. When I send it with the requests module and look at the outgoing request in Wireshark, the headers stay in the same order as provided in the dict. When I send them with httpx, the order gets modified (not alphabetic - I couldn't see any method of the ordering here)

Example (requests):
Host | 127.0.0.1:8082 Accept | */* Accept-Encoding | gzip, deflate, br User-Agent | *** Content-Type | application/json
Example (httpx):
Host | 127.0.0.1:8082 User-Agent | *** Accept-Encoding | gzip, deflate, br Accept | */* Content-Type | application/json

And when you try to overwrite the pseudo headers ( :method / :scheme / etc. ) an LocalProtocolError (Illegal header name) is raised.

[tomchristie]

When I send them with httpx, the order gets modified (not alphabetic - I couldn't see any method of the ordering here)

That's not the behaviour I'm seeing.
Sending a simple request I see...

# The headers as sent to `httpcore`...
[(b'Host', b'www.example.com'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.18.2')]

# The bytes over the wire...
b'GET / HTTP/1.1\r\nHost: www.example.com\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\nConnection: keep-alive\r\nUser-Agent: python-httpx/0.18.2\r\n\r\n'
b''
b''

[tomchristie]

At a guess you might be seeing different header ordering over HTTP/2 - I don't immediately know enough about the binary framing used to make a call on that without investing a chunk of time looking into it in a bit more detail.