fix: revert problematic channel handling from PR #1641

[sanity]

Summary

This PR addresses critical feedback from @iduartgomez on PR #1641 regarding incorrect channel usage patterns and performance issues.

Changes Made

1. Channel Handling Fixes

• Removed manual pending_connections buffer and try_send polling loop
• Converted gateway connection notification to proper async .send().await
• Kept try_send only for packet forwarding (to avoid blocking main select loop)
• Reduced channel sizes back to reasonable values (1000 → 100)

2. Rate Limiting

• Disabled bandwidth rate limiting (was set to 12.5MB/s in fix: critical production issues - decryption errors and subscription handling #1641)
• Previous implementation serialized all packets through sleep() calls, causing major throughput issues

Root Cause Analysis

The issues stemmed from a throughput mismatch:

• UDP socket receives packets quickly
• peer_connection_listener processes packets sequentially (must maintain order per connection)
• When channels filled up, packets were misrouted causing decryption failures
• This cascaded into spurious new connection attempts

What Still Needs to Be Done

Per @iduartgomez's review comments:

1. Network Subscribers: Need to ensure network-level subscriptions (tracked in OpManager/Ring) are working correctly - this is the priority
2. Application Subscribers: WebSocket client subscriptions (tracked in contract executor) may be affecting app execution, but this is secondary

As Nacho noted:

There are 2 different kind of "subscribers", the ones in the network, and the applications to the contract executor. We care about the 1st, but the 2nd may be affecting app execution, in the end a subscriber app should be receiving all updates nevertheless, so is a proxy for the 1st. But first I would focus on ensuring the 1st works correctly.

Testing

• Code compiles successfully
• Clippy warnings fixed
• Ping app tests need verification
• Network subscription functionality needs testing
• Integration tests need to be run

This is a partial fix focusing on the most critical channel handling issues. Network subscription issues will be addressed in follow-up commits.

Fixes issues raised in #1641 (review)

[iduartgomez]

Reviewed.

There still is a part I don't understand:

peer_connection_listener processes packets sequentially (must maintain order per connection)

Mmm it receives them as they are created AFAIK and internally will re-send if necessary, it does not necessarily have to receive them sequentially although it will do usually. But I don't think this is an important fact.

When channels filled up, packets were misrouted causing decryption failures

This is the part I don't understand, why channels filling up would provoke misrouting? This sounds like a mismanagement of packet routing to the individual PeerConnection since one does not follow the other. How/where we fixed this?

[sanity]

Claude: Regarding your questions about packet processing and misrouting:

1. Sequential processing: You're correct that packets don't need to be received sequentially. My description was imprecise. The issue is about processing speed, not order.

2. Channel filling causing misrouting: The misrouting happens in p2p_protoc.rs lines 242-291. When no existing connection exists for a PUT/GET/SUBSCRIBE/UPDATE operation, the code was incorrectly creating a new connection:

None => {
    tracing::warn\!("No existing outbound connection, establishing connection first");
    // This should NOT happen for PUT/GET/SUBSCRIBE/UPDATE operations\!
    self.bridge.ev_listener_tx.send(Right(NodeEvent::ConnectPeer { ... }))
}

When channels fill up during high throughput, packets meant for existing connections couldn't be queued. The system then mistakenly tried to route these packets as if they were for new connections, causing them to be handled by the wrong connection handler with different encryption keys, hence the decryption failures.

The fix ensures that PUT/GET/SUBSCRIBE/UPDATE operations only use existing connections and never trigger new connection creation.

[sanity]

Claude: I've updated the code to separate network and application subscriptions as requested by @iduartgomez. The NetworkDebugInfo structure now has:

pub struct NetworkDebugInfo {
    /// Application-level subscriptions (WebSocket clients subscribed to contracts)
    pub application_subscriptions: Vec<SubscriptionInfo>,
    /// Network-level subscriptions (nodes subscribing to contracts for routing)
    pub network_subscriptions: Vec<(ContractKey, Vec<PeerId>)>,
    pub connected_peers: Vec<PeerId>,
}

This makes debugging much clearer by explicitly separating the two types of subscriptions.

[sanity]

The test failure appears to be transient - the same test configuration passes locally. Requesting a re-run of the failed CI check.