bug: Errors in onPeerDisconnected and onPeerConnected crash application

[rmeissner]

Description

The methods onPeerDisconnected and onPeerConnected in the Subscription are async methods and are registered on the PeerManager as event handlers:

https://github.com/waku-org/js-waku/blob/52e5c34520aa98576d51e6f62a058076221f3cfe/packages/sdk/src/filter/subscription.ts#L364-L373

Due to this it is not possible to handle errors happening in these methods and the whole application crashes.

An example for this is a "race condition" that happens when 2 peers in short concession disconnect, which creates a situation where a stream cannot be created and therefore an error is thrown:

/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80
      throw new Error(
            ^


Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
    at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
    at async StreamManager.getStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:45:14)
    at async FilterCore.subscribe (/workspaces/harbour/node_modules/@waku/core/src/lib/filter/filter.ts:66:20)
    at async Subscription.requestSubscribe (/workspaces/harbour/node_modules/@waku/sdk/src/filter/subscription.ts:543:22)
    at async Promise.all (index 0)
    at async Subscription.attemptSubscribe (/workspaces/harbour/node_modules/@waku/sdk/src/filter/subscription.ts:512:21)

Expected Behavior

I would expect that it is possible to handle these errors (i.e. by providing an error handler).

Steps to Reproduce

There are no simple steps to reproduce. I was in an environment with an unstable connection. Potentially deactivating the network connection while Waku is active with multiple peers could trigger this.

Environment Details

• Version of js-waku packages: 0.0.33
• Browser/Node.js version: Node.js v22.16.0
• Operating System: Linux

Logs

waku:stream-manager:/vac/waku/peer-exchange/2.0.0-alpha1:error Failed to createStreamWithLock: Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at runNextTicks (node:internal/process/task_queues:65:5)
at listOnTimeout (node:internal/timers:549:9)
at process.processTimers (node:internal/timers:523:7)
at async StreamManager.createStreamWithLock (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:100:7) +0ms
waku:stream-manager:/vac/waku/metadata/1.0.0:error Failed to createStreamWithLock: Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at runNextTicks (node:internal/process/task_queues:65:5)
at listOnTimeout (node:internal/timers:549:9)
at process.processTimers (node:internal/timers:523:7)
at async StreamManager.createStreamWithLock (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:100:7) +0ms
waku:stream-manager:/vac/waku/store-query/3.0.0:error Failed to createStreamWithLock: Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at async StreamManager.createStreamWithLock (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:100:7) +0ms
waku:stream-manager:/vac/waku/lightpush/2.0.0-beta1:error Failed to createStreamWithLock: Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at runNextTicks (node:internal/process/task_queues:65:5)
at listOnTimeout (node:internal/timers:549:9)
at process.processTimers (node:internal/timers:523:7)
at async StreamManager.createStreamWithLock (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:100:7) +0ms
waku:stream-manager:/vac/waku/filter-subscribe/2.0.0-beta1:error Failed to createStreamWithLock: Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at runNextTicks (node:internal/process/task_queues:65:5)
at listOnTimeout (node:internal/timers:549:9)
at process.processTimers (node:internal/timers:523:7)
at async StreamManager.createStreamWithLock (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:100:7) +0ms
waku:dialer:error Error dialing peer 16Uiu2HAmSy7wsazNftaAtGiMZCnK2t4koSVrSpPZc8okFAqzPwXY DOMException [TimeoutError]: The operation was aborted due to timeout
at new DOMException (node:internal/per_context/domexception:53:5)
at Timeout._onTimeout (node:internal/abort_controller:155:9)
at listOnTimeout (node:internal/timers:588:17)
at process.processTimers (node:internal/timers:523:7) +17s
waku:dialer:info Processing dial queue: dialing 3 peers, 0 remaining in queue +2s
waku:dialer:info Dialing peer from queue: 16Uiu2HAmSy7wsazNftaAtGiMZCnK2t4koSVrSpPZc8okFAqzPwXY +0ms
waku:dialer:info Dialing peer from queue: 16Uiu2HAmSy7wsazNftaAtGiMZCnK2t4koSVrSpPZc8okFAqzPwXY +1ms
waku:dialer:info Dialing peer from queue: 16Uiu2HAmB5tV8pmbLEXynhp3avH1Jwzrioys2i4TAXzttBV998Tp +0ms
waku:health-indicator:info onPeerDisconnected: received libp2p event +30s
waku:health-indicator:info onPeerDisconnected: has connections, ignoring +0ms
waku:health-indicator:info onPeerDisconnected: node identified as Unhealthy +0ms
waku:sdk:filter-subscription:info Peer disconnected: 16Uiu2HAmG2vCWGAdp9sfdWVZPWMBqgumeiR9k6Ei2WdCL2ajn6f4 +30s
waku:sdk:filter-subscription:info Active peer 16Uiu2HAmG2vCWGAdp9sfdWVZPWMBqgumeiR9k6Ei2WdCL2ajn6f4 disconnected, removing from peers list +0ms
waku:sdk:filter-subscription:info Attempting to subscribe: useNewContentTopics=false, useOnlyNewPeers=true, contentTopics=1 +0ms
waku:peer-manager:info Getting peers for protocol: filter, pubsubTopic: /waku/2/rs/1/7 +30s
waku:connection-manager:info Getting connected peers for codec undefined +30s
waku:connection-manager:info Found 1 connected peers for codec undefined +0ms
waku:peer-manager:info Found 1 connected peers +0ms
waku:peer-manager:info Peer 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 qualifies for protocol filter +1ms
waku:peer-manager:info Found 1 locked peers out of 1 qualifying peers +0ms
waku:peer-manager:info Found 0 unlocked peers, need 1 more +0ms
waku:peer-manager:info Locking peer 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 +0ms
waku:peer-manager:info Selected 1 peers: 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 +0ms
waku:sdk:filter-subscription:info Subscribing with 1 peers for 1 content topics +1ms
waku:sdk:filter-subscription:info requestSubscribe: pubsubTopic:/waku/2/rs/1/7 contentTopics:/safe/harbour/v1/txs +0ms
waku:stream-manager:/vac/waku/filter-subscribe/2.0.0-beta1:info Attempting to create a stream for peerId=16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 multicodec=/vac/waku/filter-subscribe/2.0.0-beta1 +13s
waku:connection-limiter:info No connections, dialing peers from store +5s
waku:connection-limiter:info Dialing peers from store +0ms
waku:health-indicator:info onPeerDisconnected: received libp2p event +2ms
waku:health-indicator:info onPeerDisconnected: node identified as Unhealthy +0ms
waku:sdk:filter-subscription:info Peer disconnected: 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 +1ms
waku:metadata:error Error handling metadata request AbortError: The operation was aborted
at raceSignal (/workspaces/harbour/node_modules/race-signal/src/index.ts:51:17)
at Object.read (/workspaces/harbour/node_modules/it-byte-stream/src/index.ts:112:39)
at Object.read (/workspaces/harbour/node_modules/it-length-prefixed-stream/src/index.ts:85:41)
at read (/workspaces/harbour/node_modules/@libp2p/multistream-select/src/multistream.ts:29:28)
at Module.readString (/workspaces/harbour/node_modules/@libp2p/multistream-select/src/multistream.ts:43:21)
at Module.select (/workspaces/harbour/node_modules/@libp2p/multistream-select/src/select.ts:88:36)
at async ConnectionImpl.newStream (/workspaces/harbour/node_modules/libp2p/src/upgrader.ts:512:15)
at async ConnectionImpl.newStream (/workspaces/harbour/node_modules/libp2p/src/connection/index.ts:135:20)
at async (/workspaces/harbour/node_modules/libp2p/src/connection-monitor.ts:104:28) {
type: 'aborted',
code: 'ABORT_ERR'
} +0ms
waku:sdk:filter-subscription:info Active peer 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 disconnected, removing from peers list +3ms
waku:sdk:filter-subscription:info Attempting to subscribe: useNewContentTopics=false, useOnlyNewPeers=true, contentTopics=1 +0ms
waku:peer-manager:info Getting peers for protocol: filter, pubsubTopic: /waku/2/rs/1/7 +4ms
waku:connection-manager:info Getting connected peers for codec undefined +5ms
waku:connection-manager:info No connected peers +0ms
waku:peer-manager:info Found 0 connected peers +0ms
waku:peer-manager:info Found 0 locked peers out of 0 qualifying peers +0ms
waku:peer-manager:info Found 0 unlocked peers, need 2 more +0ms
waku:peer-manager:info Selected 0 peers: +0ms
waku:sdk:filter-subscription:info Subscribing with 0 peers for 1 content topics +0ms
waku:sdk:filter-subscription:warn Requested to use only new peers, but no peers found, skipping +30s
waku:connection-limiter:info Found 41 peers in store, and found 0 connections +8ms
waku:dialer:info Skipping peer 16Uiu2HAmRv1iQ3NoMMcjbtRmKxPuYBbF9nLYz2SDv9MTN8WhGuUU - recently failed to dial +3s
waku:dialer:info Skipping peer 16Uiu2HAmSy7wsazNftaAtGiMZCnK2t4koSVrSpPZc8okFAqzPwXY - recently failed to dial +1ms
waku:connection-limiter:info Dialing 5 peers from store +1ms
waku:dialer:info Skipping peer: 16Uiu2HAmRv1iQ3NoMMcjbtRmKxPuYBbF9nLYz2SDv9MTN8WhGuUU +0ms
waku:dialer:info Skipping peer: 16Uiu2HAmSy7wsazNftaAtGiMZCnK2t4koSVrSpPZc8okFAqzPwXY +0ms
waku:dialer:info Added peer to dialing queue, queue size: 1 +0ms
waku:dialer:info Added peer to dialing queue, queue size: 2 +0ms
waku:dialer:info Added peer to dialing queue, queue size: 3 +0ms
waku:connection-limiter:info Dialed 5 peers from store +0ms
/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80
throw new Error(
^

Error: Failed to create a new stream for 16Uiu2HAmQYiojgZ8APsh9wqbWNyCstVhnp9gbeNrxSEQnLJchC92 -- AbortError: The operation was aborted
at StreamManager.createStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:80:13)
at async StreamManager.getStream (/workspaces/harbour/node_modules/@waku/core/src/lib/stream_manager/stream_manager.ts:45:14)
at async FilterCore.subscribe (/workspaces/harbour/node_modules/@waku/core/src/lib/filter/filter.ts:66:20)
at async Subscription.requestSubscribe (/workspaces/harbour/node_modules/@waku/sdk/src/filter/subscription.ts:543:22)
at async Promise.all (index 0)
at async Subscription.attemptSubscribe (/workspaces/harbour/node_modules/@waku/sdk/src/filter/subscription.ts:512:21)

[weboko]

This should resolve the problem - #2546

Can you try using this version of @waku/sdk - 0.0.34-252e379.0?

[rmeissner]

@weboko it is kind of hard to test this as this version contain some breaking changes (as it introduces the routing concept for decoders). Do you have some migration guide prepared for this (i.e. how to create the routing info)?

[rmeissner]

I still encounter a similar error:

file:///usr/src/app/node_modules/@waku/core/dist/lib/stream_manager/stream_manager.js:53
            throw new Error(`Failed to create a new stream for ${peerId.toString()} -- ` + lastError);
                  ^

Error: Failed to create a new stream for 16Uiu2HAmNhHxVr8NiLtdQAW24VHrBi7q1pEFJpQQvdVuv7CGBWmy -- AbortError: The operation was aborted
    at StreamManager.createStream (file:///usr/src/app/node_modules/@waku/core/dist/lib/stream_manager/stream_manager.js:53:19)
    at async StreamManager.getStream (file:///usr/src/app/node_modules/@waku/core/dist/lib/stream_manager/stream_manager.js:29:18)
    at async FilterCore.subscribe (file:///usr/src/app/node_modules/@waku/core/dist/lib/filter/filter.js:29:24)
    at async Subscription.requestSubscribe (file:///usr/src/app/node_modules/@waku/sdk/dist/filter/subscription.js:320:26)
    at async Promise.all (index 0)
    at async Subscription.attemptSubscribe (file:///usr/src/app/node_modules/@waku/sdk/dist/filter/subscription.js:306:25)

Here the logs immediately before:

2025-08-14T16:30:35.389Z libp2p:connection-manager onDisconnect remove all connections for peer 16Uiu2HAmNhHxVr8NiLtdQAW24VHrBi7q1pEFJpQQvdVuv7CGBWmy
2025-08-14T16:30:35.389Z waku:connection-limiter:info No connections, dialing peers from store
2025-08-14T16:30:35.389Z waku:connection-limiter:info Dialing peers from store
2025-08-14T16:30:35.389Z waku:health-indicator:info onPeerDisconnected: received libp2p event
2025-08-14T16:30:35.389Z waku:health-indicator:info onPeerDisconnected: node identified as Unhealthy
2025-08-14T16:30:35.389Z libp2p:mplex:stream:initiator:1585 abort with error AbortError: The operation was aborted
    at raceSignal (file:///usr/src/app/node_modules/race-signal/dist/src/index.js:29:19)
    at Object.read (file:///usr/src/app/node_modules/it-byte-stream/dist/src/index.js:58:47)
    at Object.read (file:///usr/src/app/node_modules/it-length-prefixed-stream/dist/src/index.js:45:49)
    at read (file:///usr/src/app/node_modules/@libp2p/multistream-select/dist/src/multistream.js:21:30)
    at Module.readString (file:///usr/src/app/node_modules/@libp2p/multistream-select/dist/src/multistream.js:32:23)
    at Module.select (file:///usr/src/app/node_modules/@libp2p/multistream-select/dist/src/select.js:72:38)
    at async ConnectionImpl.newStream [as _newStream] (file:///usr/src/app/node_modules/libp2p/dist/src/upgrader.js:353:50)
    at async ConnectionImpl.newStream (file:///usr/src/app/node_modules/libp2p/dist/src/connection/index.js:95:24)
    at async file:///usr/src/app/node_modules/libp2p/dist/src/connection-monitor.js:48:40 {
  type: 'aborted',
  code: 'ABORT_ERR'
}
2025-08-14T16:30:35.389Z libp2p:mplex:stream:initiator:1585 try to send reset to remote
2025-08-14T16:30:35.389Z libp2p:mplex initiator stream with id 1585 and protocol undefined ended
2025-08-14T16:30:35.390Z waku:sdk:filter-subscription:info Peer disconnected: 16Uiu2HAmNhHxVr8NiLtdQAW24VHrBi7q1pEFJpQQvdVuv7CGBWmy