How to deploy ION to kubernetes?

[ahmed-adly-khalil]

Your environment.

• Version: v1.10.0
• Browser: latest chrome
• Other Information -

What did you do?

I have deployed ION to Kubernetes, everything is working except for SFU unable to create peer connection, here is my deployment and service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ion-sfu
spec:
  selector:
    matchLabels:
      app: ion-sfu
  replicas: 1
  template:
    metadata:
      labels:
        app: ion-sfu
    spec:
      imagePullSecrets:
        - name: github-registry-key
      containers:
        - name: ion-sfu
          image: pionwebrtc/ion:latest-sfu
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          ports:
            - containerPort: 3478

---
apiVersion: v1
kind: Service
metadata:
  name: ion-sfu
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 3478
      targetPort: 3478
  selector:
    app: ion-sfu
---

once trying to get video streaming working, i'm getting the below in browser

and the below in ION-SFU pod logs

[2022-01-15 06:22:51.406] [INFO] [peer.go:199] => PeerLocal got offer logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-15 06:22:51.407] [DEBUG] [publisher.go:117] => ice connection status logger=ion-sfu-node/sfu state=checking v=1
[2022-01-15 06:22:51.407] [INFO] [peer.go:210] => PeerLocal send answer logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-15 06:22:51.407] [DEBUG] [peer.go:165] => on publisher ice candidate called for peer logger=ion-sfu-node/sfu peer_id=ahmed v=1
[2022-01-15 06:22:51.407] DEBUG default: [service.go:145] [sfu.(*SFUService).Signal.func2] [S=>C] peer.OnIceCandidate: target = 0, candidate = candidate:706165708 1 udp 2130706431 45.79.247.128 5000 typ host 
[2022-01-15 06:22:51.407] DEBUG default: [service.go:238] [sfu.(*SFUService).Signal] [S=>C] join.description: answer v=0

I also updated the sfu.toml config to have my cluster public ip

nat1to1 = ["public ip"]

is there a way to know what's missing in the config?

[ahmed-adly-khalil]

@leewardbound would you be able to have a look? Thank you so much

[leewardbound]

hey, so this is a familiar issue, i think we used to have some docs/examples about this but i'm not sure where they are right now --

the issue is related to how webrtc works, basically UDP ports need to be open (but not load balanced or mapped to different host ports). this is tricky on kubernetes. in config.toml, you'll see that we have some UDP port range configured, usually 5000-5200 is the default (which allows about 200 participants i believe?). when a webrtc client (like pion or ion-sfu) sends an offer/answer packet, it includes a list of suggested IPs and ports for the connection to be opened. you need more than just nat1to1={public_ip} in k8s, because a) this doesn't open the proper ports, and b) the public IP will actually depend on which node the pod is scheduled to run on, which you can't anticipate when you are setting up config.toml.

my preferred choice to workaround this is to use a DaemonSet instead of a Deployment (assuring 1 ion-sfu is scheduled on each node), and to use HostNetwork: true so that the pod is able to a) discover its own IP address, and b) open the 5000-5200 UDP ports, without listing them 1-by-1 in the Ports section as would normally be necessary in k8s, which doesn't support opening port ranges.

i've never deployed ion (the whole app) on k8s, just ion-sfu, but here's the config i've used as recently as ion-sfu:1.9, i haven't really updated my ion-sfu projects in a while:

---
apiVersion: v1
kind: Service
metadata:
  name: sfu
  labels:
    name: sfu
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9123"
spec:
  type: ClusterIP
  ports:
    - name: grpc
      port: 50051
      targetPort: grpc
    - name: jsonrpc
      port: 7000
      targetPort: jsonrpc
  selector:
    app: sfu
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: sfu
spec:
  selector:
    matchLabels:
      name: sfu
  template:
    metadata:
      labels:
        name: sfu
        app: sfu
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      terminationGracePeriodSeconds: 30
      containers:
      - name: sfu
        image: pionwebrtc/ion-sfu:v1.9.4-allrpc
        command: ["/usr/local/bin/sfu", "-c", "/configs/sfu.toml", "-gaddr", ":50051", "-jaddr", ":7000", "-maddr", ":9123"]
        ports:
          - name: grpc
            containerPort: 50051
          - name: jsonrpc
            containerPort: 7000
          - name: metrics
            containerPort: 9123

[ahmed-adly-khalil]

@leewardbound Thank you so much, i tried deploying ion-sfu beside the rest of ion pods and didn't get much luck, maybe because both sfu app/container inside ion has different deplyoment requirements from the independent ion-sfu.

currently i'm getting the below 2 lines in both signal and sfu pods and trying to wrap my head around, hopefully tonight.

[2022-01-16 21:54:48.548]  INFO nats-discovery.Client: [client.go:91] [client.(*Client).handleNatsMsg] node.delete: dc1.sfu-EhR9Kq 
[2022-01-16 21:54:48.548]  INFO default: [node.go:138] [ion.(*Node).handleNeighborNodes] Service down: rtc node id => [sfu-EhR9Kq] 

one observation that the docker version that works locally has the following ports setup

  sfu:
    image: pionwebrtc/ion:latest-sfu
    build:
      dockerfile: ./docker/sfu.Dockerfile
      context: .
    volumes:
      - "./configs/docker/sfu.toml:/configs/sfu.toml"
    ports:
      - "5000:5000/udp"
      - 3478:3478

which means, theoretically at least, it can work with only 3478 and 5000 if configured correctly. That's why i'm trying to get the following service ports to work:

apiVersion: v1
kind: Service
metadata:
  name: sfu
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      name: tcp
      port: 3478
      targetPort: 3478
    - protocol: UDP
      name: udp
      port: 5000
      targetPort: 5000
  selector:
    app: sfu

I will continue digging and post my findings.

@Sean-Der @adwpc by chance do you have ideas around here?

Thank you everyone so much

[ahmed-adly-khalil]

comparing the sfu logs between k8s on linode and local docker:

k8s on linkdoe:

[2022-01-16 23:39:59.718] [INFO] [peer.go:223] => PeerLocal got answer logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.718] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:3056717505 1 udp 2122262783 2001:1970:5692:c200:e93e:77c8:131f:ca35 57113 typ host generation 0 ufrag 8pjS network-id 2 network-cost 10 
[2022-01-16 23:39:59.718] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.718] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=checking v=1
[2022-01-16 23:39:59.768] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:4081163164 1 udp 1685987071 24.150.74.40 49344 typ srflx raddr 192.168.0.2 rport 49344 generation 0 ufrag 8pjS network-id 1 network-cost 10 
[2022-01-16 23:39:59.768] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.815] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:4172427825 1 tcp 1518283007 2001:1970:5692:c200:e93e:77c8:131f:ca35 9 typ host tcptype active generation 0 ufrag 8pjS network-id 2 network-cost 10 
[2022-01-16 23:39:59.815] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:39:59.816] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:345893049 1 tcp 1518214911 192.168.0.2 9 typ host tcptype active generation 0 ufrag 8pjS network-id 1 network-cost 10 
[2022-01-16 23:39:59.817] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0
[2022-01-16 23:40:00.871] DEBUG default: [service.go:346] [sfu.(*SFUService).Signal] [C=>S] description: offer v=0
o=- 562677169881253040 3 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1
a=extmap-allow-mixed
a=msid-semantic: WMS 6d704ff9-279a-41e4-b1dd-f9d0b802c1e9
m=application 54207 UDP/DTLS/SCTP webrtc-datachannel

local docker

 

[2022-01-16 23:38:21.028] [INFO] [peer.go:223] => PeerLocal got answer logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.029] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=checking v=1

[2022-01-16 23:38:21.029] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:3056717505 1 udp 2122262783 2001:1970:5692:c200:e93e:77c8:131f:ca35 59848 typ host generation 0 ufrag 9P/6 network-id 2 network-cost 10 

[2022-01-16 23:38:21.030] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.030] DEBUG default: [service.go:418] [sfu.(*SFUService).Signal] [C=>S] trickle: target 0, candidate candidate:1511920713 1 udp 2122194687 192.168.0.2 61493 typ host generation 0 ufrag 9P/6 network-id 1 network-cost 10 

[2022-01-16 23:38:21.031] [INFO] [peer.go:243] => PeerLocal trickle logger=ion-sfu-node/sfu peer_id=ahmed v=0

[2022-01-16 23:38:21.126] [DEBUG] [subscriber.go:58] => ice connection status logger=ion-sfu-node/sfu state=connected v=1

[2022-01-16 23:38:22.403] DEBUG default: [service.go:346] [sfu.(*SFUService).Signal] [C=>S] description: offer v=0

o=- 8127501359152418552 3 IN IP4 127.0.0.1

the docker version the ICE switch to connected right after checking while on k8s this doesn't happen. I tried also using STUN server with no luck, proceeding with digging ...

[ahmed-adly-khalil]

looking at the following offer and answer, what i can see both has 127.0.0.1 which i anticipate to be the issue, still tring to find ways around that

{
    "type": "offer",
    "sdp": "v=0\r\no=- 2711940938522023088 3 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=extmap-allow-mixed\r\na=msid-semantic: WMS 3a8fffb3-cf06-44f8-a10c-144e209991bb\r\nm=application 57934 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 24.150.74.40\r\na=candidate:3194784207 1 udp 2122262783 2001:1970:5692:c200:bc83:7b6d:313e:f371 61453 typ host generation 0 network-id 2 network-cost 10\r\na=candidate:1511920713 1 udp 2122194687 192.168.0.2 57934 typ host generation 0 network-id 1 network-cost 10\r\na=candidate:4081163164 1 udp 1685987071 24.150.74.40 57934 typ srflx raddr 192.168.0.2 rport 57934 generation 0 network-id 1 network-cost 10\r\na=candidate:4042224959 1 tcp 1518283007 2001:1970:5692:c200:bc83:7b6d:313e:f371 9 typ host tcptype active generation 0 network-id 2 network-cost 10\r\na=candidate:345893049 1 tcp 1518214911 192.168.0.2 9 typ host tcptype active generation 0 network-id 1 network-cost 10\r\na=ice-ufrag:0/zJ\r\na=ice-pwd:+9L5W9JtXWOfH+NRU7kzFlHr\r\na=ice-options:trickle\r\na=fingerprint:sha-256 F6:D4:47:2E:60:11:71:0A:47:E4:66:B0:8E:AB:9B:67:03:BD:FB:56:0F:F7:93:A9:8E:8F:50:3B:A6:DB:64:36\r\na=setup:actpass\r\na=mid:0\r\na=sctp-port:5000\r\na=max-message-size:262144\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:0/zJ\r\na=ice-pwd:+9L5W9JtXWOfH+NRU7kzFlHr\r\na=ice-options:trickle\r\na=fingerprint:sha-256 F6:D4:47:2E:60:11:71:0A:47:E4:66:B0:8E:AB:9B:67:03:BD:FB:56:0F:F7:93:A9:8E:8F:50:3B:A6:DB:64:36\r\na=setup:actpass\r\na=mid:1\r\na=extmap:1 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 urn:3gpp:video-orientation\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=sendonly\r\na=msid:3a8fffb3-cf06-44f8-a10c-144e209991bb 2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=ssrc:2659409037 cname:TiUOWZgap2jBYNbL\r\na=ssrc:2659409037 msid:3a8fffb3-cf06-44f8-a10c-144e209991bb 2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\na=ssrc:2659409037 mslabel:3a8fffb3-cf06-44f8-a10c-144e209991bb\r\na=ssrc:2659409037 label:2dd8c802-f7be-4801-a6aa-df98e21c13c8\r\n"
}




{
    "type": "answer",
    "sdp": "v=0\r\no=- 3940562069726591883 1642473100 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1\r\na=msid-semantic: WMS\r\na=ice-lite\r\nm=application 5000 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 45.79.247.128\r\na=candidate:706165708 1 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=candidate:706165708 2 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=candidate:706165708 1 udp 2130706431 45.79.247.128 5000 typ host generation 0\r\na=ice-ufrag:NGrczxsIXdECNyjl\r\na=ice-pwd:dEenZietQjixEhwubLiIGYBsXjueWNHA\r\na=fingerprint:sha-256 90:EA:F3:AC:B3:3E:B7:30:D4:EB:DC:86:F5:55:0B:CC:F8:1C:E7:2C:B3:5B:2E:B4:F4:EF:3C:89:DF:44:80:A7\r\na=setup:active\r\na=mid:0\r\na=sctp-port:5000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:NGrczxsIXdECNyjl\r\na=ice-pwd:dEenZietQjixEhwubLiIGYBsXjueWNHA\r\na=fingerprint:sha-256 90:EA:F3:AC:B3:3E:B7:30:D4:EB:DC:86:F5:55:0B:CC:F8:1C:E7:2C:B3:5B:2E:B4:F4:EF:3C:89:DF:44:80:A7\r\na=setup:active\r\na=mid:1\r\na=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\n"
}