Skip to content

Connection System Design

Jump to bottom
Xiang Hong edited this page Aug 18, 2024 · 4 revisions

WARNNING This document is still working in progress

In DualDrill Engine, we are facing problems including

how to pass information between client and server

Since we are developing a library to make it easier to support distributed rendering and user inputs, we need to find a unified abstraction and proper implementation for distributed message passing. There are lots of candidate technologies, we need to find those ones which meets following creteria:

  • Usablity in browser, since our first-class client is browser
  • Usablity in XR devices, since we are also planning to support XR headset
  • Bi-directional real time video sending/receving support, which also requires P2P connection in client to client scenarios for better performance
  • Bi-directional message channel support, since we want to abstract complex usages like editors into client, bi-directional message is required
  • Large data transfer support

As a result, we choose the combination of the following two technologies:

  • WebRTC as main protocol for message and video, since its support in browser and there exists some implementations on server side
  • Use Websocket for WebRTC signal service
  • HTTP for large data (e.g. mesh, texutures, volumn data, etc), for server to client large data push, use WebRTC message to trigger a client request like a call back (the easiest way is sending a message with a proper RESTful url for the resource)

Note: for XR devices, if we are planning to use WebXR apis, then will work since we are already considering browser support. But since currently most XR applications are developed using Unity, we need to ensure the WebRTC/Websocket and HTTP client implementation is supported.

Note: Signal server can be decoupled with other program logic as long as we can pass proper connection identity information to it. All signal server's need is to establish three channels (offer, anwser, icecandidate), those channels only need low bandwith and can tolerate high latency. We are currently using WebSocket for that.

Considered candidates:

  • SignalR, great support for simple bi-directional message, with request-response pattern support. But it does not support video. Does not work in Unity.
  • Blazor server interactive, greate for user input via HTML UI, support client-server bi-directional message (actually over underlying SignalR connection), does not support video, low performance for high frequency events like user mouse inputs, does not work in Unity.
  • Websocket, good support for bi-directional message, need investigation on Unity support (for specific XR devices)

WebRTC Server Side Implementation Problem

In server side, we may change WebRTC implementation (SIPSorcery, WebView2, libdatachannel, etc). SIPSorcery is the solution with most references, but it suffers for high GC pressure. Currently we use WebView2 as a workaround. We should investigate libdatachannel or libcef bindings as better solutions.

It would be better if we can decouple connection related systems to render related systems, we are planning to achieve this by using a combination of HeadlessSurface's Capture method, MediaStream abstraction and IPeerConnection abstraction to achieve this.

Clone this wiki locally