No stall/freeze signal when render loop stops receiving frames

[zoharbabin]

Why

The render loop is driven entirely by requestVideoFrameCallback (or requestAnimationFrame as fallback). If the underlying track goes silent while readyState/paused/ended still look normal — a real WebRTC failure mode (e.g. the remote peer's encoder hangs, or a network path drops without tearing down the connection) — the callback simply stops firing. renderFrame() never gets called again, the canvas silently freezes on the last frame, and the page has no way to detect this happened. There is no timeout, watchdog, or event that surfaces a stall.

What (confirmed in code)

The library's full emitted event surface (core, excluding the generic plugin emit() passthrough) is exactly:

Event	Location
error	src/chromakey.js:672, src/chromakey.js:1136
started	src/chromakey.js:752
autotune	src/chromakey.js:888
backend	src/chromakey.js:1079
pluginerror	src/chromakey.js:994-996

None of these fire in response to a stalled/frozen stream — error only fires on the native video element's error event or an internal render/setup exception, not on frame delivery stopping.

The render loop (_startLoop, src/chromakey.js:1199-1215) uses requestVideoFrameCallback (preferred, lines 1200/1204/1206) or requestAnimationFrame (fallback, lines 1208-1213, gated on !video.paused && !video.ended). Neither path tracks elapsed wall-clock time between successive callback invocations, so there's no way to notice "the last frame callback was N seconds ago and nothing new has arrived."

Confirmed no existing stall-detection primitive anywhere in the codebase:

• No watchdog, no setTimeout at all in the file (the only timer is a setInterval inside the plugin API's every(ms, fn) helper, src/chromakey.js:954, which is a generic facility for plugin authors, not used internally for stall detection).
• readyState is checked in several places (lines 678, 679, 739, 827, 867, 1255) but only as a one-shot "is it ready enough to render this call" gate, never compared across time to detect that it stopped changing.
• MediaStreamTrack.readyState is never referenced anywhere in src/.
• The requestVideoFrameCallback metadata argument (which carries presentation/media timestamps usable for stall detection) is discarded — the callback (src/chromakey.js:1201) takes no parameters at all.

How (suggested fix)

• Track the timestamp of the last successful frame callback.
• Run a lightweight watchdog (e.g. setInterval, or a check piggybacked on the existing render loop) that compares "now" against the last-frame timestamp. If it exceeds a threshold (configurable, with a sane default — e.g. 2-3 seconds) while the video is not paused/ended, dispatch a new event, e.g. stalled, with detail including the elapsed time.
• Optionally dispatch a paired recovered (or similar) event once frames resume after a stall was flagged, so integrators can clear any "reconnecting" UI they showed.
• Make the threshold configurable via the existing options object, consistent with how other tunables are exposed.
• Don't confuse this with the native waiting/stalled events on the <video> element — those fire based on buffering state and are not reliable for a live WebRTC track that reports readyState/paused as healthy while frames have actually stopped.

Success criteria

• A test (or manual repro) that freezes frame delivery on a live track (readyState/paused/ended unaffected) and asserts a stalled event fires within the configured threshold, with useful detail.
• No stalled event fires during normal playback, intentional pause(), or ended — only on unexpected frame-delivery interruption.
• Threshold is documented and configurable; default value is called out in the README.

[zoharbabin]

Phase 1 spec (harnessed-build)

Root cause (established in the issue body): the render loop only calls renderFrame() when requestVideoFrameCallback/requestAnimationFrame fire, and the only emitted events are backend, started, error, autotune, pluginerror — none of which fire when frame delivery silently stops while readyState/paused/ended still look healthy (the real WebRTC failure mode this issue is about).

Design decision, stated up front so Phase 3 doesn't relitigate it: the reliable stall signal is video.getVideoPlaybackQuality().totalVideoFrames (a monotonic decoded-frame counter, supported Chrome/Firefox/Safari 15.4+), checked on a separate low-frequency watchdog interval — not video.currentTime. For a live MediaStream source, currentTime generally advances on the stream's own real-time clock regardless of whether new frames are actually being decoded, so it would produce false negatives on exactly the failure mode this issue describes. totalVideoFrames only increments when a frame is actually decoded/presented, and checking it off the hot path costs nothing extra in _startLoop()/renderFrame().

#	Category	Rule / N/A	Proof
1	Isolation	Rule: stall-tracking state (_lastFrameQualityCount, _stallStartedAt, _isStalled) lives on this (the instance), and the watchdog setInterval handle is per-instance and cleared in destroy().	New e2e: 2 concurrent instances, stall one, assert the other is unaffected and its own watchdog interval is independently cleaned up on destroy (no leaked timer).
2	Security	N/A — no new I/O, no string built from untrusted input, no dynamic execution. Detail payload on the stalled/recovered events is numeric (elapsedMs, frame counts) only.	`grep -RInE 'eval(
3	Resiliency	N/A — this fix does not perform network I/O; it observes local decoder-frame counters.	N/A, no gate needed.
4	Performance	Rule: the watchdog runs on its own setInterval at a fixed, bounded cadence — it must not add any per-frame work inside _startLoop()/renderFrame() (the render hot path). Rule: getVideoPlaybackQuality is feature-detected; if unsupported, the watchdog is a no-op (no thrown error, no false events) rather than falling back to an unreliable signal.	Code review confirms no new call from _startLoop()/renderFrame(); e2e test confirms no stalled/error event is ever thrown when getVideoPlaybackQuality is stubbed to undefined.
5	Clean code	Rule: one _checkStall() method, called from one setInterval, started in the constructor and cleared in destroy(). New stallTimeout option is live-updatable via update() (not added to IMMUTABLE_OPTIONS) and exposed on the custom element as a stall-timeout attribute in ELEMENT_OPTION_ATTRIBUTES, following the existing min-key/fade-top pattern. Class-level JSDoc event list updated to include stalled/recovered.	Code review + lint-equivalent (node --check) + doc comment diff.
6	Compliance	N/A — no PCI/PII/audit-relevant data; the feature only inspects local decoder frame counters.	N/A, no gate needed.

Fix direction (not a drafted implementation):

• Add stallTimeout to DEFAULTS (ms; a value <= 0 disables the watchdog). Default TBD during implementation, empirically chosen to avoid false positives on normal brief buffering.
• Constructor starts one setInterval watchdog (cadence independent of stallTimeout, e.g. checked a few times per timeout window); destroy() clears it.
• Watchdog: skip while video.paused || video.ended; otherwise compare getVideoPlaybackQuality().totalVideoFrames to the last-seen count. No change for >= stallTimeout ms → dispatch stalled (once, with { elapsedMs } detail) until frames resume, at which point dispatch recovered.
• Custom element: no separate implementation — it forwards to the same ChromaKeyVideo instance's events; only needs the new attribute mapping.

Empirical validation required before implementation is trusted (Phase 2): confirm in this exact CI environment (headless Chromium + swiftshader) that getVideoPlaybackQuality().totalVideoFrames actually stops incrementing when a canvas.captureStream() (draw-triggered, no explicit fps) source's redraw loop is stopped, and resumes incrementing when redraws resume — this is the whole test harness's foundation, so it gets verified empirically, not assumed.

[zoharbabin]

Phase 4 — Termination

Implemented in #5. All gates pass.

#	Category	Rule	Gate	Result
1	Isolation	All new state (_lastFrameQualityCount, _stallStartedAt, _isStalled, _stalledElapsedMs, _stallWatchdog) lives on this, no module-level mutable state	e2e: "stalling one instance does not affect another" (2 concurrent instances, stall one, assert the other never fires)	✅ pass
2	Security	No dynamic code execution, no untrusted input reaches a sink	`grep -RInE "eval(	new Function(
3	Resiliency	Watchdog treats paused/ended as healthy (resets timer, no false 'stalled'); disableable via stallTimeout <= 0	e2e: "does not fire during uninterrupted playback", "does not fire while intentionally paused", "stallTimeout <= 0 disables detection"	✅ pass (3/3)
4	Performance	Fixed, cheap polling cadence (STALL_CHECK_INTERVAL_MS = 500ms), one setInterval per instance, cleared in destroy()	code review; bundle size gate	✅ +64 lines source, minified 22.7KB/24KB budget, gzip 7.4KB/8KB budget
5	Clean code	One _checkStall() implementation, JSDoc on the method and the two new event types, no dead code/TODOs	code review; node --check	✅ pass
6	Compliance	N/A — no new I/O, no new data storage/transmission, no PII touched	—	N/A

Design decision, tested empirically before implementation: uses video.getVideoPlaybackQuality().totalVideoFrames rather than video.currentTime. For a live MediaStream, currentTime advances on the stream's own real-time clock regardless of whether a new frame was actually decoded — it would miss exactly the stall this feature exists to catch. Verified with a throwaway diagnostic test in this repo's actual CI environment (headless Chromium + swiftshader) that frame-count freezes exactly during a simulated stall and resumes after, before writing the real implementation.

Deviation from spec: none. The stallTimeout option/stall-timeout attribute and 'stalled'/'recovered' events match the original issue body as written.

PR: #5 (CI green: https://github.com/kaltura/chroma-key-video/actions/runs/32535032413). Not merged — left open per instruction for manual review.