Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ext/node): support createConnection option in node:http.request() #25470

Merged
merged 143 commits into from
Dec 13, 2024

Conversation

satyarohith
Copy link
Member

@satyarohith satyarohith commented Sep 5, 2024

Work in progress. Feature works but breaks existing tests.

Closes #19507

@satyarohith satyarohith force-pushed the support_create_connection branch from c807b0e to abda518 Compare September 11, 2024 05:33
@satyarohith satyarohith force-pushed the support_create_connection branch from 89a4de0 to e98525c Compare September 24, 2024 05:32
@kt3k kt3k force-pushed the support_create_connection branch from aedab15 to 9c1b39b Compare September 26, 2024 03:45
@kt3k kt3k force-pushed the support_create_connection branch from bbd15fb to 2f9cdaa Compare September 26, 2024 04:05
@kt3k kt3k added the ci-draft Run the CI on draft PRs. label Sep 26, 2024
@kt3k kt3k changed the title WIP: support createConnection option in node:http.request() fix(ext/node): WIP support createConnection option in node:http.request() Sep 26, 2024
@bartlomieju
Copy link
Member

Possibly closes #26182

@0xkalle
Copy link

0xkalle commented Dec 10, 2024

I am also ready to test with #26735 as soon as I should.

Copy link
Member

@bartlomieju bartlomieju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bartlomieju bartlomieju merged commit 960776c into denoland:main Dec 13, 2024
17 checks passed
@satyarohith satyarohith deleted the support_create_connection branch December 13, 2024 05:28
@blouflashdb
Copy link

I this released yet?

@0xkalle
Copy link

0xkalle commented Dec 15, 2024

I'd guess as it is on main you can use canary to try it. But as far as I know it's not in a release yet.

@bartlomieju
Copy link
Member

It's not released yet. please run deno upgrade canary and let us know if it solved the problem for you.

@0xkalle
Copy link

0xkalle commented Dec 15, 2024

#26735 fixed on canary.

@StreetStrider
Copy link

canary + ws works for me now. Thank you for work 👍

Unix domain socket is not working, though, while they're working in the same node's test. Not sure why, it seem to not create socket file.

@bartlomieju
Copy link
Member

Thanks for the feedback, I'm glad to hear that you got unblocked folks!

Unix domain socket is not working, though, while they're working in the same node's test. Not sure why, it seem to not create socket file.

@StreetStrider could you open a separate issue about this one? Having a specific repro you are using would be so much helpful 🙏

@StreetStrider
Copy link

StreetStrider commented Dec 18, 2024

@bartlomieju here's the repro
I may be missing something, but I think listening to unix socket is enough for a file to occur. At least that's how it worked in my Node.js+ws test.
deno-sock.tar.gz

@blouflashdb
Copy link

works now on canary

dsherret pushed a commit that referenced this pull request Jan 9, 2025
…#25470)

This commit changes "node:http" module to add support
for the "createConnection" option when the "request()"
API is called.


Closes #19507

---------

Signed-off-by: Yoshiya Hinosawa <[email protected]>
Signed-off-by: Satya Rohith <[email protected]>
Co-authored-by: Yoshiya Hinosawa <[email protected]>
Co-authored-by: Bartek Iwańczuk <[email protected]>
Co-authored-by: crowlkats <[email protected]>
kt3k added a commit that referenced this pull request Jan 13, 2025
#27639)

See the comment
#25470 (comment) for
the reason why we do this workaround to make `make-fetch-happen` work in
Deno

This PR applies the same workaround to `npm-check-updates` package.
`npm-check-updates` internally uses
[`npm-registry-fetch`](https://www.npmjs.com/package/npm-registry-fetch)
which uses
[`make-fetch-happen`](https://www.npmjs.com/package/make-fetch-happen)
(the problematic package) for making http request to npm registry.

The detection of `make-fetch-happen` doesn't work for
`npm-check-updates` because we use call stack at `net.Socket`
constructor to check if it's called from `make-fetch-happen`, but
`npm-check-updates` bundles its dependency and the check doesn't work.

This PR adds the check of `npm-check-updates` string in call stack in
net.Socket constructor to trigger the workaroud.

closes #27629
@mariusheil
Copy link

mariusheil commented Jan 15, 2025

I just tested Deno 2.1.5 (the first release to include the changes) but apparently this update has broken Mqtt.js (tested version 5.10.1) in some way. While I was previously getting a successful connect with Deno 2.1.4, it is now failing reproducibly with Deno 2.1.5. I have switched versions back and forth a few times to verify this.

Summary:

  • This PR seems to affect Mqtt.js (e.g. 5.10.1) with Websockets
  • MQTT websocket connect works in Deno 2.1.4 but broke in Deno 2.1.5
  • Does not seem to affect mqtts (ssl) transport
  • Can be verified against https://test.mosquitto.org/
    • Use host:"test.mosquitto.org", port:8091, path:"/ws", protocol:"wss", username:"rw", password:"readwrite"

Working in Deno 2.1.4:

MqttService                          INFO  2025-01-15T12:14:38.820Z: MQTT Connecting....
  mqttjs connecting to an MQTT broker... +0ms
  mqttjs:client MqttClient :: version: 5.10.1 +0ms
  mqttjs:client MqttClient :: environment node +0ms
  mqttjs:client MqttClient :: options.protocol wss +1ms
  mqttjs:client MqttClient :: options.protocolVersion 4 +0ms
  mqttjs:client MqttClient :: options.username Token-MYTOKEN +0ms
  mqttjs:client MqttClient :: options.keepalive 60 +0ms
  mqttjs:client MqttClient :: options.reconnectPeriod 10000 +0ms
  mqttjs:client MqttClient :: options.rejectUnauthorized false +1ms
  mqttjs:client MqttClient :: options.properties.topicAliasMaximum undefined +0ms
  mqttjs:client MqttClient :: clientId Token-MYTOKEN__1736943278820 +0ms
  mqttjs:client MqttClient :: setting up stream +0ms
  mqttjs:client connect :: calling method to clear reconnect +1ms
  mqttjs:client _clearReconnect : clearing reconnect timer +0ms
  mqttjs:client connect :: using streamBuilder provided to client to create stream +0ms
  mqttjs calling streambuilder for wss +21ms
  mqttjs:ws streamBuilder +0ms
  mqttjs:ws createWebSocket +1ms
  mqttjs:ws protocol: MQTT 4 +0ms
  mqttjs:ws creating new Websocket for url: wss://myexampleserver:443/ws and protocol: mqtt +0ms
Warning: Not implemented: ClientRequest.options.createConnection
  mqttjs:client connect :: pipe stream to writable stream +3ms
  mqttjs:client connect: sending packet `connect` +0ms
  mqttjs:client _writePacket :: packet: {
  mqttjs:client   cmd: 'connect',
  mqttjs:client   protocolId: 'MQTT',
  mqttjs:client   protocolVersion: 4,
  mqttjs:client   clean: true,
  mqttjs:client   clientId: 'Token-MYTOKEN__1736943278820',
  mqttjs:client   keepalive: 60,
  mqttjs:client   username: 'Token-MYTOKEN',
  mqttjs:client   password: 'myPass',
  mqttjs:client   properties: undefined
  mqttjs:client } +1ms
  mqttjs:client _writePacket :: emitting `packetsend` +1ms
  mqttjs:client _writePacket :: writing to stream +0ms
  mqttjs:client _writePacket :: writeToStream result true +28ms
MetadataService                      INFO  2025-01-15T12:14:39.086Z: Orga Name: Marius
  mqttjs:client writable stream :: parsing buffer +671ms
  mqttjs:client parser :: on packet push to packets array. +2ms
  mqttjs:client work :: getting next packet in queue +1ms
  mqttjs:client work :: packet pulled from queue +0ms
  mqttjs:client _handlePacket :: emitting packetreceive +0ms
  mqttjs:client _handleConnack +1ms
  mqttjs:client _setupKeepaliveManager :: keepalive 60 (seconds) +0ms
  mqttjs:client KeepaliveManager: set keepalive to 60000ms +0ms
  mqttjs:client connect :: sending queued packets +2ms
  mqttjs:client deliver :: entry undefined +0ms
  mqttjs:client _resubscribe +0ms
MqttService                          INFO  2025-01-15T12:14:39.552Z: MQTT Connected

I am now getting a disconnect after a very short time:

Broken in Deno 2.1.5:

MqttService                          INFO  2025-01-15T12:11:46.451Z: MQTT Connecting....
  mqttjs connecting to an MQTT broker... +0ms
  mqttjs:client MqttClient :: version: 5.10.1 +0ms
  mqttjs:client MqttClient :: environment node +0ms
  mqttjs:client MqttClient :: options.protocol wss +0ms
  mqttjs:client MqttClient :: options.protocolVersion 4 +1ms
  mqttjs:client MqttClient :: options.username Token-MYTOKEN +0ms
  mqttjs:client MqttClient :: options.keepalive 60 +0ms
  mqttjs:client MqttClient :: options.reconnectPeriod 10000 +0ms
  mqttjs:client MqttClient :: options.rejectUnauthorized false +0ms
  mqttjs:client MqttClient :: options.properties.topicAliasMaximum undefined +0ms
  mqttjs:client MqttClient :: clientId Token-MYTOKEN__1736943106451 +0ms
  mqttjs:client MqttClient :: setting up stream +0ms
  mqttjs:client connect :: calling method to clear reconnect +1ms
  mqttjs:client _clearReconnect : clearing reconnect timer +0ms
  mqttjs:client connect :: using streamBuilder provided to client to create stream +0ms
  mqttjs calling streambuilder for wss +12ms
  mqttjs:ws streamBuilder +0ms
  mqttjs:ws createWebSocket +0ms
  mqttjs:ws protocol: MQTT 4 +0ms
  mqttjs:ws creating new Websocket for url: wss://myexampleserver.io:443/ws and protocol: mqtt +0ms
  mqttjs:client connect :: pipe stream to writable stream +16ms
  mqttjs:client connect: sending packet `connect` +1ms
  mqttjs:client _writePacket :: packet: {
  mqttjs:client   cmd: 'connect',
  mqttjs:client   protocolId: 'MQTT',
  mqttjs:client   protocolVersion: 4,
  mqttjs:client   clean: true,
  mqttjs:client   clientId: 'Token-MYTOKEN__1736943106451',
  mqttjs:client   keepalive: 60,
  mqttjs:client   username: 'Token-MYTOKEN',
  mqttjs:client   password: 'myPass',
  mqttjs:client   properties: undefined
  mqttjs:client } +0ms
  mqttjs:client _writePacket :: emitting `packetsend` +1ms
  mqttjs:client _writePacket :: writing to stream +0ms
  mqttjs:client _writePacket :: writeToStream result true +28ms
MetadataService                      INFO  2025-01-15T12:11:46.733Z: Orga Name: Marius
  mqttjs:client streamErrorHandler :: error Bad resource ID +317ms
  mqttjs:client noop :: BadResource: Bad resource ID
    at node:http:306:27
    at HttpsClientRequest._writeHeader (node:http:398:7)
    at HttpsClientRequest._flushHeaders (node:_http_outgoing:382:12)
    at TLSSocket.onConnect (node:http:444:16)
    at TLSSocket.emit (ext:deno_node/_events.mjs:405:35)
    at _afterConnect (node:net:159:12)
    at _afterConnectMultiple (node:net:214:3)
    at TCP.afterConnect (ext:deno_node/internal_binding/connection_wrap.ts:43:11)
    at TCP.handle.afterConnect (node:_tls_wrap:157:29)
    at eventLoopTick (ext:core/01_core.js:175:7) {
  name: 'BadResource'
} +0ms
  mqttjs:client (Token-MYTOKEN__1736943106451)stream :: on close +1ms
  mqttjs:client _flushVolatile :: deleting volatile messages from the queue and setting their callbacks as error function +0ms
  mqttjs:client stream: emit close to MqttClient +0ms
  mqttjs:client close :: connected set to `false` +1ms
  mqttjs:client close :: clearing connackTimer +0ms
  mqttjs:client close :: calling _setupReconnect +0ms
  mqttjs:client _setupReconnect :: emit `offline` state +0ms
  mqttjs:client _setupReconnect :: set `reconnecting` to `true` +0ms
  mqttjs:client _setupReconnect :: setting reconnectTimer for 10000 ms +0ms
MqttService                          WARN  2025-01-15T12:11:46.829Z: MQTT Disconnected

I have seen that it was connecting once after a few retries, but did work a lot better previously.

Marius

@kt3k
Copy link
Member

kt3k commented Jan 15, 2025

@mariusheil Thanks for your report! Could you share some minimal script to reproduce the issue?

I'd also appreciate if you would open a new issue about that

@mariusheil
Copy link

Hi, I have opened #27694 as a bug report.

bartlomieju pushed a commit that referenced this pull request Jan 16, 2025
#27639)

See the comment
#25470 (comment) for
the reason why we do this workaround to make `make-fetch-happen` work in
Deno

This PR applies the same workaround to `npm-check-updates` package.
`npm-check-updates` internally uses
[`npm-registry-fetch`](https://www.npmjs.com/package/npm-registry-fetch)
which uses
[`make-fetch-happen`](https://www.npmjs.com/package/make-fetch-happen)
(the problematic package) for making http request to npm registry.

The detection of `make-fetch-happen` doesn't work for
`npm-check-updates` because we use call stack at `net.Socket`
constructor to check if it's called from `make-fetch-happen`, but
`npm-check-updates` bundles its dependency and the check doesn't work.

This PR adds the check of `npm-check-updates` string in call stack in
net.Socket constructor to trigger the workaroud.

closes #27629
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-draft Run the CI on draft PRs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

node: ClientRequest.options.createConnection
10 participants