Record every frame of a batched emit and downmix stereo audio

[aviksethia99]

Why

A model that emits media in multi-frame batches hands Recorder.on_chunk a whole generation window at once — one call can carry a full second of frames plus its audio. The feed queue between the emit thread and the encoder held four frames, and on the first queue.Full the loop abandoned the rest of the chunk. Because a Python tight loop out-races the encoder thread for the GIL, exactly the head of every burst survived: a session recorded as a ~7× time-lapse of itself, deterministically. Measured end to end with the real encoder, a 12-second batched session recorded 1.65 seconds of media; the same fix brings it to 12.02 seconds with zero drops. Per-frame emitters never fill the queue and are unaffected (6.02s recorded for a 6s session, before and after).

Separately, a stereo (2, M) audio track was flattened channel-after-channel into the jitter buffer, so recordings carried alternating stretches of the left channel and the right channel instead of the mix. With a constant L=+8000/R=−8000 test signal the decoded recording averaged |s|≈7847 (raw channel blocks); after the fix it averages 0 (the correct mix).

What Changed

The queue is sized to absorb a two-second burst on the recording grid (_FEED_QUEUE_MAX_FRAMES = 2 * RECORDING_FPS). The sizing argument lives where the constant does: on_chunk queues the entire resampled burst before the emit thread moves on to the connection pacers, and the pacers then pay the same media out in real time — which is exactly the window the encoder has to drain the queue. So the queue never needs to be "large", it needs to hold one burst, and the non-blocking contract of on_chunk (documented at the call site in the runner) stays intact. An overflowing slot now drops alone (continue) rather than taking the rest of the chunk with it (break); the drop counter and its rate-limited warning are unchanged, and a dropped slot discards its already-pulled audio so the tracks stay aligned.

_buffer_audio now reduces the track through to_int16_mono — the reduction the live transport applies on the WebRTC path — instead of a blind reshape(-1). Multi-channel audio mixes down per sample, mono passes through untouched, and float input picks up the correct int16 scaling as a side effect of reusing the helper. Importing a transport.webrtc helper from recording is the one layering wrinkle in the diff; frames.py is pure NumPy with no libwebrtc dependency, but if the panel prefers, the helper could move to a shared home in a follow-up.

Two tests pin the behaviours: a batched emit (24 frames at fps=24 → 30 grid frames) must queue every grid frame with the feed worker parked, and a stereo buffer must yield the per-sample mix rather than a run of one channel.

🤖 Generated with Claude Code