Embedded media decoder and player (like mpv)

[john-basilio]

Is there a good way to receive, decode, and render raw h264 video data from tcp?

please ask specifics its my first time tackling this

[Lncvrt]

I am not sure exactly what you want but I imagine using ffmpeg in rust would make this possible.

[MukundaKatta]

Yes, this is very doable in a Tauri app. The main decision is where you decode. Three sane architectures, cheapest to most flexible:

1. Remux in Rust, decode in the webview

The fastest path if your H.264 is reasonably well-formed. Take the raw bytes off TCP, repackage them as fragmented MP4 (fMP4), and feed them to a standard <video> element via MediaSource Extensions. The webview already has a hardware-accelerated H.264 decoder on every platform Tauri supports — don't reinvent it.

• Rust side: use mp4 or mp4-atom to build init + media segments, or ffmpeg-next / rsmpeg with the mp4 muxer if the stream needs reordering.
• Delivery: either
  • Command::emit the Uint8Array chunks to the webview and sourceBuffer.appendBuffer(...) them, or
  • Spin up a tiny local HTTP server (axum, actix-web, or Tauri's own custom protocol handler) and point <video>.src at it. Custom protocol is the cleaner choice — no port collision, no CORS, and you get per-window scoping.

This is the path most "live video in Tauri" apps end up on. CPU cost is basically muxing overhead; decoding is GPU-accelerated by the webview.

2. Decode in Rust, render in the webview

If the source isn't standards-compliant (odd NAL unit boundaries, missing SPS/PPS, custom framing), decode it yourself and ship raw frames.

• Decoder: openh264, ffmpeg-next, or on macOS/iOS you can wrap VideoToolbox via av-foundation.
• Transport: convert YUV → RGBA, encode to JPEG (image, turbojpeg) or as raw bytes, send via Window::emit and draw to <canvas>.
• Be aware this is CPU-heavy at 1080p30 — Base64/JSON serialization in the IPC bridge will dominate. Use tauri::ipc::Response with a byte slice, or route via a custom protocol so the payload doesn't round-trip through JSON.

3. Native overlay (skip the webview)

If you need the absolute lowest latency and smoothest playback (telemetry, low-latency camera, gaming), render into a separate native surface and position a transparent webview over it for UI chrome. gstreamer-rs with playbin + an NSView/HWND overlay gets you there, but you lose the nice "everything in one WebView" property. Usually only worth it below ~150 ms latency budgets.

Concrete starter (option 1)

// src-tauri/src/main.rs — sketch
use tauri::{Manager, http::{Response, Request, ResponseBuilder}};
use tokio::sync::broadcast;

#[tauri::command]
async fn start_stream(state: tauri::State<'_, broadcast::Sender<Vec<u8>>>) -> Result<(), String> {
    // connect to TCP, read H.264 Annex-B NAL units, mux to fMP4,
    // push each segment into the broadcast channel
    Ok(())
}

fn main() {
    let (tx, _rx) = broadcast::channel::<Vec<u8>>(32);

    tauri::Builder::default()
        .manage(tx.clone())
        .register_uri_scheme_protocol("stream", move |_app, _req| {
            // stream fMP4 to <video src="stream://live">
            // use chunked transfer and flush on every segment
            let body = /* pull from broadcast rx */ vec![];
            ResponseBuilder::new()
                .header("Content-Type", "video/mp4")
                .body(body)
        })
        .invoke_handler(tauri::generate_handler![start_stream])
        .run(tauri::generate_context!())
        .expect("error while running tauri application");
}

<video src="stream://live" autoplay muted playsinline></video>

If you can share (a) whether the TCP source is already in a container or is raw Annex-B, and (b) your latency budget, I can narrow the suggestion further.