Using http.request undocumented behavior

[yotamN]

I noticed that in the chain method within chain.ts, http.request is being used with headers as a string array instead of an object. This behavior is undocumented and could potentially cause issues in the future. Currently, OpenTelemetry auto-instrumentation is affected because it expects headers to be an object.

Is there a specific reason for using an array for headers? If not, I have opened a pull request to address this issue.

[jancurn]

The reason is that some servers or clients duplicate HTTP headers and we want to proxy the headers as they are, to reduce a chance of blocking. For details, see https://nodejs.org/docs/latest/api/http.html#responsewriteheadstatuscode-statusmessage-headers, where this behavior is documented

[yotamN]

I understand your point. However, in the case of a CONNECT request, isn't the order of headers only relevant to the proxy server itself? This means that I cannot change it in forward mode. Did your research conclude that this is the only way to order the headers?

Additionally, regarding the documentation, you referred to the documentation for response.writeHead. The documentation for http.request still does not mention the handling of headers as an array.

I will proceed to fix this in OpenTelemetry as well, since you are likely not the only library affected. However, it is important to note that this behavior could technically change in any future version.

[jancurn]

Oh you're right, I thought this is an external interface, but here in this case it make sense to change this as this is internal usage. Thanks for your help.

[yotamN]

Thank you.
I also opened an issue with NodeJS and it seems someone already opened a PR in there as well.

[jirimoravcik]

Fixed in #584

[tonybruess]

It seems like forward is also passing headers as an array which breaks Charles and Brightdata proxies. Can I open a PR to fix?

[jancurn]

breaks Charles and Brightdata proxies

How does it break them? You mean remote proxies or some local ones?

[tonybruess]

Example of array headers breaking charles:

Versus passing headers as an object:

[yotamN]

Do you use OpenTelemetry or any other instrumentation library?
Your first example is a clear case where the array is accidentally being converted into an object, but not by proxy-chain.
I have an open PR in OpenTelemetry that I still work on but I'm not sure when I will have the time to finish it.

[tonybruess]

I use DataDog but am not currently attempting to capture these requests. I am experiencing issues with using anonymizeProxy with a brightdata residential proxy.

[yotamN]

I use DataDog but am not currently attempting to capture these requests. I am experiencing issues with using anonymizeProxy with a brightdata residential proxy.

I have no experience with DataDog or any other instrumentation that you might run, I can only assure you that passing an array is a valid usage of the http.request function. You can look at the implementation here.

[jirimoravcik]

I agree with @yotamN.
Could you please try the same code without any instrumentation libraries, if you can still reproduce it, please provide a minimal reproduction example, thanks.