URL normalization removes equals sign from empty query parameters

[zeroc00I]

httpx version:

v1.7.1

Current Behavior:

When httpx receives a URL with a query parameter that has an empty value (e.g., https://example.com/?param=), it normalizes the URL by stripping the trailing equals sign (=) before making the request. The actual HTTP request is sent to https://example.com/?param.

Expected Behavior:

httpx should preserve the trailing equals sign (=) for query parameters with empty values. If the input URL is https://example.com/?param=, the HTTP request should also be made to https://example.com/?param=.

Steps To Reproduce:

Run the following command to pipe a URL with an empty-value parameter to httpx with request debugging enabled:

echo "https://example.com/?test=" | httpx -debug-req

Observe the HTTP request in the debug output. You will see that the request is made to https://example.com/?test, without the trailing =.

Anything else:

This URL normalization, while seemingly minor, changes the semantic meaning of the request for many common backend technologies, which can prevent the discovery of certain vulnerabilities or misconfigurations.

The server-side interpretation of ?key versus ?key= is inconsistent across platforms:

?key=: This is universally interpreted as a key (key) with an empty string value ("").

?key: This is often interpreted as a key with a null value, particularly by frameworks like ASP.NET (on IIS) and Ruby on Rails. Other technologies (like PHP or Node.js) may treat it as an empty string.

In my specific use case, I was testing an ASP.NET application that behaves differently when a parameter is null versus an empty string (""). Because httpx removed the =, my test case failed as the server received null instead of the empty string I intended to send.

Preserving the trailing = is crucial for ensuring that the user's intent is accurately reflected in the outgoing request, allowing for precise security testing across diverse backend environments.

I believe this will make a big difference in discovering bugs that were overlooked by the community, and I thank Project Discovery for all the tools they have made available as open source and free of charge.