Video Input Streaming for Flight Director

[mdonahoo]

Is your feature request related to a problem? Please describe.

As a Flight Director, I want to display a live-video feed to the players, so that I can interact with them in real-time as a character to tell the story.

Describe the solution you'd like

The Flight Director can stream a video/audio feed from a webcam, HDMI input, or computer screen to the Viewscreen. The video feed will allow the Flight Director to interact with the players as a character or part of the story.

The Flight Director can start and stop the stream, as well as mute and turn off video. The video steam only shows on the Viewscreen when it is being streamed, the players cannot control it.

Additional context

The Flight Director can stream their webcam, or in my case, stream an HDMI input from another computer which is rendering, with audio, a live-action Unreal metahuman so that I can use a virtual character instead of my webcam.

Here's an example of the streaming in my fork of the codebase. Viewscreen on the left, and Flight Director core Video screen on the right.

[mdonahoo]

Overview

Thorium Nova implements real-time video streaming from the Flight Director to multiple
Viewscreen clients using WebRTC peer-to-peer connections with a signaling server for
coordination.

Architecture

Server-Side (/app/.server/data/videoStream.ts)

The server manages streaming state and facilitates WebRTC signaling between clients:

• Stream Management: Tracks active streams, source devices, and target viewscreens
• Signaling Relay: Routes WebRTC offers, answers, and ICE candidates between peers
• Subscription System: Provides real-time updates to clients about stream status

Key endpoints:

• status - Subscribable endpoint for stream state updates
• start/stop - Control stream lifecycle
• signaling - Send signaling messages
• signalingSubscription - Subscribe to receive signaling messages

Flight Director Side

VideoStreamCore (/app/cores/VideoStreamCore/index.tsx)

• UI for device selection and target viewscreen management
• Initiates streaming sessions with selected clients
• Displays local preview of the stream

useWebRTC Hook (/app/cores/VideoStreamCore/useWebRTC.ts)

• Creates and manages WebRTC peer connections
• Sends offers to target viewscreens
• Subscribes to signaling messages to receive answers and ICE candidates
• Handles the complete WebRTC negotiation process

Viewscreen Client Side

VideoReceiverFixed (/app/cards/Viewscreen/VideoReceiverFixed.tsx)

• Subscribes to stream status updates
• Receives WebRTC offers through the signaling subscription
• Creates answers and establishes peer connections
• Renders the received video stream

Data Flow

1. Stream Initialization

• Flight Director selects target viewscreens and starts streaming
• Server notifies selected viewscreens about incoming stream
• Each viewscreen prepares to receive the connection

2. WebRTC Connection Establishment

• Flight Director creates peer connections and sends offers
• Viewscreens receive offers, create answers, and send them back
• Both sides exchange ICE candidates for NAT traversal
• Direct peer-to-peer connections established

3. Media Streaming

• Video and audio flow directly between Flight Director and Viewscreens
• No media data passes through the server
• Multiple simultaneous connections supported

Key Design Patterns

• Live Query Subscriptions: Real-time updates using the framework's subscription system
• Targeted Message Routing: Signaling messages filtered by clientId
• Separation of Concerns: Clear distinction between UI, WebRTC logic, and server
  communication
• Type Safety: Full TypeScript implementation with proper type definitions

Implementation Highlights

The system leverages the existing live query infrastructure for efficient real-time
communication while maintaining clean separation between the signaling plane (through
the server) and the media plane (direct peer-to-peer). This architecture ensures
scalability and low latency for video streaming across multiple simulator stations.

[alexanderson1993]

A few things to be thoughtful of:

• How does the Flight Director play multiple roles?
• How does this work with the Flight Director's microphone? Would it be in the shot?
• What does the Flight Director use for backgrounds? Costumes?
• How is background audio handled? What if the Flight Director has some staff to direct at the same time as being on the video call?
• How does the crew interact back with the Flight Director? If the crew are on a single bridge, we can assume the Flight Director has a camera and microphone on the bridge to see and hear the crew - what about flights over the internet?
• How does the affect the viewscreen being displayed on multiple individual crew stations? (it will be possible to render the viewscreen in a picture-in-picture window on an arbitrary number of stations)
• Do Flight Directors actually want this, or would they be satisfied with a "theater of the mind" when talking over the viewscreen?

That last point might have some bigger architectural implications for flights over the internet with multiple players spread out some distance from each other. Does the Flight Director computer make individual connections to each player's viewscreen? Or do we need to set up some kind of SFU? Would it be necessary to use a third-party service to route the video calls in a performant way?

Lots of things to figure out with this, and video calls aren't necessary to run flights, so this feature is low on my priority list.

I don't intend to add any realtime AV support to Thorium Nova, at least not for a long time. Discord & friends can handle that stuff for the time being.

[mdonahoo]

I thought this feature would align great with the future vision of the software:
https://spacecenter.alpineschools.org/donate/

Your questions make it seem like you've never used a camera to tell a story. It doesn't always have to be the flight director that plays the part, it can be another cast member.

As a flight director, I would use green screens, and costumes to pretend to be an engineer or fleet admiral. I'm a gamer, so I have a special setup using Unreal metahumans to puppet over video.

I only vibe coded this in a couple days for proof of concept. I won't code this if you don't intend this to be a feature.

[alexanderson1993]

That link that you provide is for a space center that isn't currently affiliated with Thorium Nova development. Their roadmap and our roadmap aren't the same.

[mdonahoo]

Is there another Space Center you're working with to know what features they want? To help inform your new roadmap?

[alexanderson1993]

No, I'm not working on any space centers at this time - just the community, which they participate in sometime, and my own wishes for the software.

I started on Thorium Classic in 2016. Before that, I started some bridge simulator software called Flint in 2014. I made four sets of simulator controls for the CMSC between 2006 and 2011.

Thorium Nova is the culmination of all of that, including all the features I wished I could build into those controls but couldn't. In other words, the last 20 years of my experiences working at and with space centers is what's informing the roadmap.