routr WS behind a TLS-offloading load balancer

[SemiConscious]

Hi All.

I have routr-one set up and working behind an EC2 load balancer, as an ECS task. When processing WSS, the load balancer does the TLS and contacts routr on a non-TLS websocket. Everything goes fine through the INVITE and the media starts flowing. After this, things don't go so well - UPDATEs appear to be sent to rtpengine and then disappear, and BYEs cause routr to crash, and from the logs it looks as if it's trying to generate TLS packets. Is there some setup I need to do? I have tried setting up the client with a contact_uri with transport=ws but this makes no difference.

Sample UPDATE from client end:

UPDATE sip:[email protected]:53843;transport=ws SIP/2.0
Route: <sip:3.11.175.88:5063;lr;transport=wss>;r2=on
Route: <sip:3.11.175.88:5062;lr;transport=ws>;r2=on
Via: SIP/2.0/WSS avatar.natterbox-dev05.net;branch=z9hG4bK8463636
Max-Forwards: 69
To: <sip:[email protected]>;tag=gtok28vm0r
From: "1002" <sip:[email protected]>;tag=iuk0kdhgbe
Call-ID: 8cheg9bfsedfscu5su35
CSeq: 7666 UPDATE
Contact: <sip:[email protected];transport=ws;ob>
Session-Expires: 90;refresher=uac
Allow: INVITE,ACK,CANCEL,BYE,UPDATE,MESSAGE,OPTIONS,REFER,INFO,NOTIFY
Supported: timer,ice,outbound
User-Agent: JsSIP 3.10.1
Content-Length: 0

Sample BYE (triggered by lack of response to UPDATE):

BYE sip:[email protected]:53843;transport=ws SIP/2.0
Route: <sip:3.11.175.88:5063;lr;transport=wss>;r2=on
Route: <sip:3.11.175.88:5062;lr;transport=ws>;r2=on
Via: SIP/2.0/WSS avatar.natterbox-dev05.net;branch=z9hG4bK1000715
Max-Forwards: 69
To: <sip:[email protected]>;tag=gtok28vm0r
From: "1002" <sip:[email protected]>;tag=iuk0kdhgbe
Call-ID: 8cheg9bfsedfscu5su35
CSeq: 7667 BYE
Reason: SIP ;cause=408; text="Request Timeout"
Allow: INVITE,ACK,CANCEL,BYE,UPDATE,MESSAGE,OPTIONS,REFER,INFO,NOTIFY
Supported: outbound
User-Agent: JsSIP 3.10.1
Content-Length: 0

routr log from start of BYE message:

2024-09-03T08:30:32.386+01:00 | 2024-09-02 07:30:32.385 [verbose]: (dispatcher) sending request to middleware {"ref":"8cheg9bfsedfscu5su35","middlewareRef":"rtprelay-middleware","addr":"localhost:51903","postProcessor":true,"runPostProcessorMiddlewares":false}
-- | --
  | 2024-09-03T08:30:32.386+01:00 | 2024-09-02 07:30:32.386 [verbose]: (dispatcher) forwarding request to processor {"ref":"connect-processor"}
  | 2024-09-03T08:30:32.388+01:00 | 2024-09-02 07:30:32.388 [verbose]: (connect) connect processor received new request {"ref":"8cheg9bfsedfscu5su35","method":"BYE","type":"(request)","edgePort":"ip-172-17-34-176.eu-west-2.compute.internal"}
  | 2024-09-03T08:30:32.392+01:00 | 2024-09-02 07:30:32.392 [verbose]: (dispatcher) sending request to middleware {"ref":"8cheg9bfsedfscu5su35","middlewareRef":"rtprelay-middleware","addr":"localhost:51903","postProcessor":true,"runPostProcessorMiddlewares":true}
  | 2024-09-03T08:30:32.399+01:00 | 2024-09-02 07:30:32.399 [verbose]: (rtprelay) request {"direction":"web-to-web","isNewCall":false,"type":"BYE"}
  | 2024-09-03T08:30:32.465+01:00 | 2024-09-02 07:30:32.465 [verbose]: (dispatcher) sending request to middleware {"ref":"8cheg9bfsedfscu5su35","middlewareRef":"rtprelay-middleware","addr":"localhost:51903","postProcessor":true,"runPostProcessorMiddlewares":false}
  | 2024-09-03T08:30:32.466+01:00 | 2024-09-02 07:30:32.466 [verbose]: (dispatcher) forwarding request to processor {"ref":"connect-processor"}
  | 2024-09-03T08:30:32.470+01:00 | 2024-09-02 07:30:32.470 [verbose]: (connect) connect processor received new request {"ref":"8cheg9bfsedfscu5su35","method":"BYE","type":"(response)","edgePort":"ip-172-17-34-176.eu-west-2.compute.internal"}
  | 2024-09-03T08:30:32.473+01:00 | 2024-09-02 07:30:32.473 [verbose]: (dispatcher) sending request to middleware {"ref":"8cheg9bfsedfscu5su35","middlewareRef":"rtprelay-middleware","addr":"localhost:51903","postProcessor":true,"runPostProcessorMiddlewares":true}
  | 2024-09-03T08:30:32.477+01:00 | 2024-09-02 07:30:32.476 [verbose]: (dispatcher) found messageType to be responseType and broke the chain {"ref":"8cheg9bfsedfscu5su35","middlewareRef":"rtprelay-middleware","addr":"localhost:51903"}
  | 2024-09-03T08:30:33.773+01:00 | Exception in thread "EventScannerThread" java.lang.StackOverflowError
  | 2024-09-03T08:30:33.774+01:00 | at java.base/sun.security.provider.SHA.implReset(Unknown Source)
  | 2024-09-03T08:30:33.774+01:00 | at java.base/sun.security.provider.DigestBase.engineReset(Unknown Source)
  | 2024-09-03T08:30:33.774+01:00 | at java.base/sun.security.provider.DigestBase.engineUpdate(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/java.security.MessageDigest$Delegate.engineUpdate(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/java.security.MessageDigest.update(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/sun.security.provider.SecureRandom.engineNextBytes(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/sun.security.provider.NativePRNG$RandomIO.implNextBytes(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/sun.security.provider.NativePRNG.engineNextBytes(Unknown Source)
  | 2024-09-03T08:30:33.775+01:00 | at java.base/java.security.SecureRandom.nextBytes(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.RandomCookie.<init>(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.ClientHello$ClientHelloMessage.<init>(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.ClientHello$ClientHelloKickstartProducer.produce(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.SSLHandshake.kickstart(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.ClientHandshakeContext.kickstart(Unknown Source)
  | 2024-09-03T08:30:33.776+01:00 | at java.base/sun.security.ssl.TransportContext.kickstart(Unknown Source)
  | 2024-09-03T08:30:33.777+01:00 | at java.base/sun.security.ssl.SSLEngineImpl.writeRecord(Unknown Source)
  | 2024-09-03T08:30:33.777+01:00 | at java.base/sun.security.ssl.SSLEngineImpl.wrap(Unknown Source)
  | 2024-09-03T08:30:33.777+01:00 | at java.base/sun.security.ssl.SSLEngineImpl.wrap(Unknown Source)
  | 2024-09-03T08:30:33.777+01:00 | at java.base/javax.net.ssl.SSLEngine.wrap(Unknown Source)
  | 2024-09-03T08:30:33.777+01:00 | at gov.nist.javax.sip.stack.SSLStateMachine.wrap(SSLStateMachine.java:101)
  | 2024-09-03T08:30:33.777+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.sendMessage(NioTlsWebSocketMessageChannel.java:164)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.onNewSocket(NioTlsWebSocketMessageChannel.java:365)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.NioTcpMessageChannel.sendTCPMessage(NioTcpMessageChannel.java:346)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.NioWebSocketMessageChannel.sendNonWebSocketMessage(NioWebSocketMessageChannel.java:107)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.access$501(NioTlsWebSocketMessageChannel.java:49)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel$2.doSend(NioTlsWebSocketMessageChannel.java:168)
  | 2024-09-03T08:30:33.778+01:00 | at gov.nist.javax.sip.stack.SSLStateMachine.wrap(SSLStateMachine.java:127)
  | 2024-09-03T08:30:33.779+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.sendMessage(NioTlsWebSocketMessageChannel.java:164)
  | 2024-09-03T08:30:33.779+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.onNewSocket(NioTlsWebSocketMessageChannel.java:365)
  | 2024-09-03T08:30:33.779+01:00 | at gov.nist.javax.sip.stack.NioTcpMessageChannel.sendTCPMessage(NioTcpMessageChannel.java:346)
  | 2024-09-03T08:30:33.779+01:00 | at gov.nist.javax.sip.stack.NioWebSocketMessageChannel.sendNonWebSocketMessage(NioWebSocketMessageChannel.java:107)
  | 2024-09-03T08:30:33.779+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel.access$501(NioTlsWebSocketMessageChannel.java:49)
  | 2024-09-03T08:30:33.780+01:00 | at gov.nist.javax.sip.stack.NioTlsWebSocketMessageChannel$2.doSend(NioTlsWebSocketMessageChannel.java:168)
  | 2024-09-03T08:30:33.780+01:00 | at gov.nist.javax.sip.stack.SSLStateMachine.wrap(SSLStateMachine.java:127)
... until stack is all used ...

Related question: how to get routr to quit on a crash like this? Currently it sits doing nothing. It would be great if whatever process that crashed would restart, or I could get some signal that would allow me to restart the container.

[SemiConscious]

I have modified my container to use this edgeport.yaml config:

kind: EdgePort
apiVersion: v2beta1
ref: default
metadata:
  region: default
spec:
  unknownMethodAction: Discard
  processor:
    addr: localhost:51901
  methods:
  - REGISTER
  - MESSAGE
  - INVITE
  - ACK
  - BYE
  - CANCEL
  - UPDATE
  transport:
  - protocol: tcp
    port: 5060
  - protocol: udp
    port: 5060
  - protocol: ws
    port: 5062

INVITEs now fail with this line in the log:

2024-09-03T12:31:47.874+01:00 | 2024-09-02 11:31:47.874 [error]: (connect) no listening point found for transport WSS {}
-- | --

Which I'm guessing corresponds to the SIP/2.0/WSS parameter in the INVITE VIa header:

INVITE sip:[email protected] SIP/2.0
Via: SIP/2.0/WSS avatar.natterbox-dev05.net;branch=z9hG4bK1237564
Max-Forwards: 69
To: <sip:[email protected]>
From: "1002" <sip:[email protected]>;tag=ud1rsuuj1g
Call-ID: j105p43heghp0e0r9rvk
CSeq: 427 INVITE
Authorization: Digest algorithm=MD5, username="1002", realm="avatar.natterbox-dev05.net", nonce="753769db98d1c3c5bb12fcd05324b2d3", uri="sip:[email protected]", response="7f852320745326ef0166f5fea74f0867", qop=auth, cnonce="fu0ho484kbg1", nc=00000001
Contact: <sip:[email protected];transport=ws;ob>
Content-Type: application/sdp
Allow: INVITE,ACK,CANCEL,BYE,UPDATE,MESSAGE,OPTIONS,REFER,INFO,NOTIFY
Supported: ice,replaces,outbound
User-Agent: JsSIP 3.10.1
Content-Length: 7154

Is a nice workaround possible in routr, or do I need to hack headers in the load balancer?

[SemiConscious]

I have worked around the problem by generating a certificate, but I'd love to know how this could be done.

[psanders]

I have a few thoughts about this but found myself stuck in travels. I will share my ideas as soon as soon as I get a moment.

[SemiConscious]

Thank you!

…

On Wed, 4 Sept 2024, 04:06 Pedro Sanders, ***@***.***> wrote: I have a few thoughts about this but found myself stuck in travels. I will share my ideas as soon as soon as I get a moment. — Reply to this email directly, view it on GitHub <#292 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANMOPUFTGBZU3NSCITT5P3ZUZ2MPAVCNFSM6AAAAABNRRQT22VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJTHE4DMNA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[psanders]

Routr is confused because it receives messages with WSS in the Via header it wasn't expecting. This is a bug, and we should fix Routr to either recognize a proxy in front of it or introduce a flag to assist with the routing.

If you write a bug report, I can draft a solution.