High end-to-end latency on non-B200 GPUs

[ohMargin]

I found that when running on GPUs other than B200, the actual network/display latency becomes significantly higher.

Is there a way to optimize this for lower-end GPUs? Also, would it be possible to lower the FPS or reduce the number of frames processed per step to improve real-time responsiveness and reduce accumulated latency?

[Dai-Wenxun]

@ohMargin Both of your questions got real answers since you filed this:

1. Non-B200 optimization — a full optimization round landed on main: FP8 GEMMs, fused CUDA ops (joyomni_ops), SageAttention and whole-chunk CUDA-graph capture, all on by default in deploy/run_server.sh — the deployment docs were overhauled around it (DEPLOYMENT.md), so that one script is now the single entry point. On an RTX PRO 6000 (Blackwell) this sustains 480p @ 24 fps and 720p @ 16 fps end-to-end (B200: 720p @ 30 fps). You can feel the actual latency yourself in the browser demo (runs on PRO 6000-class hardware): https://huggingface.co/spaces/wxDai/joyai-video-edit — note it runs the 480p @ 24 fps config to stay real-time, while the model is trained at 720p, so visual and editing quality in the demo are a notch below what a local 720p run gives. On the consumer-card front, RTX 5090 (32 GB) support lands this week.

2. Lower FPS / fewer frames per step — yes:

• capture FPS is a first-class knob (--fps, also exposed in the web UI); lowering it cuts compute demand proportionally,
• --num-inference-steps 1 (default 2) trades some quality for ~1.5× throughput,
• the sender is credit-gated against the server's ack stream, so latency stays bounded instead of accumulating — and a recent fix moved all per-session warmup (graph capture etc.) before the first frame is accepted, which removed the several-second latency ramp you'd previously see at session start.

If a specific GPU still underperforms after latest main, post the model + logs here.

[Dai-Wenxun]

@ohMargin Following up on the consumer-card line from my last comment: RTX 5090 (32 GB) support landed — real-time 840 × 480 @ 24 FPS on a single card:

JOYOMNI_SAGE_ATTN=1 JOYOMNI_FP8_FAST_ACCUM=1 JOYOMNI_LOW_VRAM=1 \
bash deploy/run_server.sh

The two GeForce-specific switches are latency work, exactly in the spirit of this issue: INT8 SageAttention and fp16-accumulate FP8 GEMMs both exist because GeForce runs fp32-accumulate tensor ops at half rate. Combined with the block-wise FP8 weight load (JOYOMNI_LOW_VRAM=1), the 5090 is now the cheapest hardware that runs the full pipeline in real time. Details in the Deployment Guide (RTX 5090 section).

[stayinalive181]

Some measurements from running this model on two non-B200 cards, in case they help size expectations. Per 8-frame chunk (VAE encode + DiT + decode), two steps, whole-chunk CUDA graph on:

Card	840×480	1248×720	How measured
B200	—	226 ms	paper
H100 80GB SXM	257 ms	510 ms	stock demo server, late August, FP8 with the sm_90a build from #27
RTX PRO 6000 Blackwell 96GB	253 ms FP8 / 458 ms bf16	684 ms	runtime driven in-process at 114a1f6, 2026-09-05; 720p from the stock server on 2026-08-29

What the runs established:

1. FP8 is the lever on these cards, and it is easy to run without it. On the PRO 6000, FP8 takes the DiT from 340 ms to 186 ms per chunk (capacity 23.5 → 43 fps at 480p) and cuts camera-frame-to-pixels latency from 645 / 775 ms to 433 / 564 ms (p50 / p90); resident memory 56.8 → 41.8 GiB. FP8 is enabled only through JOYOMNI_FP8_IMG / JOYOMNI_FP8_TXT, which run_server.sh exports; anyone embedding the runtime without those variables silently runs bf16 while has_fp8() reports True. A config-level switch, or a log line at load stating the effective precision, would prevent that.
2. Once the camera outruns the chunk rate, latency is queue depth, not compute. At bf16 the PRO 6000 has 23.5 fps of capacity against a 24 fps camera, so the five stage queues (four chunks deep each) fill and worker-side input age settled at 1.3 s with the graph and 3.8 s eager. Bounding the chunks in flight to three brought that to about 0.8 s p50 at the same fps. --max-inflight-chunks now covers this; the other fix is FP8 or a 20 fps camera so the card has headroom.
3. --kv-reset-frames 600 shows up as a ~1.2 s freeze every ~25 s (session rebuild). With it at 0 we ran 90 s sessions with no visible degradation on our content, only a slow palette drift. Worth documenting as the trade-off it is.
4. H100 caveat. The chunk time above was steady on every H100 we tried, but on one box the stock server settled at 9–12 fps with 60–65 % of camera frames dropped at ingest while the GPU sat ~90 % idle, i.e. the input governor feeding back on itself after a cold start rather than the card. That was before --max-inflight-chunks existed; if it still reproduces I can write it up separately.
5. H100 needs the sm_90a build (Hopper: build joyomni_ops for sm_90a, not sm_90 (FP8 GEMM aborts at launch on H100) #27); compiled as plain sm_90 the FP8 GEMM aborts at every launch and the server exits during warmup.

Our latency probe and per-stage timers are ours, not the demo UI, so end-to-end numbers on your web client will differ; the per-chunk times should transfer.

[samuellucky2424-afk]

hello Good day Never mind I just want to ask, are we still expecting the new release this month as you promised early this month So I just want to know, If we can keep waiting or we should just try other models while watching