Camera pre-roll via an Arm lifecycle phase - #1844
Open
martien-wdy wants to merge 2 commits into
Open
Conversation
Give camera capture a pre-roll so a triggered episode's video starts BEFORE the trigger, making a campaign's `buffer` mean what it says for the camera and not only for application records. Problem: today the data manager keeps a continuously maintained pre-roll ring, but it holds only application records; camera capture begins AT the trigger, so the frame that caused the episode is never recorded. The design's Arm lifecycle phase was never built (the debt register lists "No fragment pre-roll"). Cause: the camera capture adapter had no standby state. It subscribed to the producer hub and opened its episode files only when the campaign triggered, so there was no keyframe-aligned buffer of the seconds before the trigger to flush. Solution: add an Arm (standby) phase to the capture-adapter contract. A campaign that requests a buffer arms its continuous-mode camera sources: each subscribes to the device hub as a NON-owning consumer (an empty stream request, exactly like the sensor path, so it never takes a stream from a viewer and never registers an explicit-parameter holder that would force the parameter-precedence takeover from the hub-state work this stacks on) and keeps a keyframe-aligned encoded ring of `buffer` seconds. On trigger the ring is flushed into the opening camera frames at their real negative-offset canonical times, on the same subscription that then continues live. The ring reuses the application ring's byte-eviction discipline and the adapter's existing keyframe logic (frameRandomAccess), and only ever evicts whole groups of pictures so a flushed clip decodes. Frames carry the bracketed CLOCK_BOOTTIME receipt the hub stamped at broadcast and their real sample identities, so payload-retention accounting and the sample-id/model-input correlation treat them as captured frames. The manifest reports requested versus achieved pre-roll honestly: achieved is at least the buffer at steady state but shorter when armed too recently, and the episode is never failed and frames are never fabricated. Audio and ROS 2 still start at the trigger this round and report their achieved offset honestly.
…ters Pre-roll arms a standby subscription that asserts no stream parameters, so a camera source that sets BOTH a buffer and an explicit max_resolution or rate records at the producer's running parameters instead. The episode manifest already reported this honestly, but only after the fact: the operator learned about it by reading the manifest of an episode already recorded at the wrong parameters. That is the same class of silent downgrade the parameter-precedence work this stacks on exists to prevent, arriving through a different path. Warn at deployment, where the operator can still change the plan, following the existing deploy-warning pattern (unimplemented capture modes, retention.local_quota). The warning names each offending source and its explicit values, states that pre-roll and explicit stream parameters are mutually exclusive in this release, and says the values will be reported as requested but not achieved. Only sources that are actually armed conflict. Snapshot-mode camera sources are never armed, so their explicit parameters are still honored through the normal capture join and they are deliberately not named. The runtime behavior is unchanged: arming must not assert parameters, both because it would take a running camera from a viewer at deploy time for a campaign that may never trigger, and because asserting them at the trigger would change the sequence parameter set part way through a clip and cost the episode its browser-playable rendering. The campaign documentation for capture.buffer now states the mutual exclusivity.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A campaign that sets
buffer: 10spromises a clip that runs from ten secondsbefore the trigger to
after_triggerpast it. Today that promise holds only forapplication records: the data manager keeps a continuously maintained pre-roll
ring, but it walks
storedApplicationRecordonly. Camera capture begins AT thetrigger, so the frame that caused the episode is never recorded. In the demo the
subject was already in frame rather than walking in. The design's Arm lifecycle
phase for exactly this was never built and is recorded in the debt register as
"No fragment pre-roll (Arm lifecycle unimplemented)".
Cause
The camera capture adapter had no standby state. It joined the producer hub and
opened its episode files (segments,
index.jsonl,clock_samples.jsonl) onlywhen the campaign triggered, so there was no keyframe-aligned buffer of the
seconds before the trigger to flush into the episode.
Solution
Add an Arm (standby) phase to the capture-adapter contract, camera only this
round.
Where the phase lives.
dataArmingAdapteringo/internal/agent/services/data_service.gois the optional pre-rollextension of the existing capture-adapter contract (
Arm/Disarm). Thecamera adapter implements it; audio and ROS 2 do not. The episode's campaign
key is threaded to the adapter through a new
data.CaptureSession.CampaignKey, so the adapter's ownStartfinds andconsumes the ring the campaign armed.
Arming is non-owning. A parameter-less
StreamVideoRequestjoins thedevice hub exactly like the sensor path, so arming never takes a running
stream from a viewer and registers no explicit-parameter holder. That is the
bit the parameter-precedence work this branch stacks on pivots the takeover
on, so arming can never force the takeover, and if an unrelated
explicit-parameter capture restarts the producer, the armed subscriber
reattaches (like the sensor path) and marks a segment boundary in the ring.
The ring reuses existing mechanisms.
cameraPreRollRing(
data_camera_preroll.go) mirrors the application ring's byte-evictiondiscipline (
preRollBytes/evictPreRoll): drop the oldest until within thewindow and the byte cap. It is specialised so it only ever begins on a
random-access unit and only evicts whole groups of pictures, reusing the
adapter's
frameRandomAccesskeyframe logic, because a flushed clip that doesnot start on a keyframe will not decode. One ring per campaign source, bounded
by
bufferseconds and by a fixed byte ceiling, whichever is tighter.Real capture times and identities. On trigger the ring is flushed into the
opening frames on the same subscription, which then continues live. Each
buffered frame carries the bracketed
CLOCK_BOOTTIMEreceipt the hub stampedat broadcast (
receiptBootNanos) and its realsample_id, soindex.jsonlgets true negative-offset
canonical_episode_nanosand payload-retentionaccounting and the sample-id/model-input correlation treat them as captured
frames with their real ids and offsets.
Pre-roll and explicit stream parameters are mutually exclusive, and
deployment says so. Because arming asserts no parameters, a camera source
that sets both a
bufferand an explicitmax_resolutionorraterecordsboth its pre-roll and its live tail at the producer's running parameters. The
manifest reports those values as requested but not achieved, and
wendy data campaign deploynow warns at deploy time, naming each offendingsource, so the operator learns while they can still change the plan rather
than after an episode was recorded at the wrong parameters. Snapshot-mode
camera sources are never armed, so their explicit parameters are still
honored and they are not warned about.
Achieved versus requested, honestly. The manifest source entry keeps the
requested offset (
-buffer) and reports the achieved offset from how far backthe ring actually reached: at steady state at least the buffer, but shorter
right after arming. If the ring cannot reach a keyframe at or before
X - buffer, it flushes from the earliest keyframe it has and recordsachieved < requested. It never fabricates frames and never fails the episode.
Audio and ROS 2 still start at the trigger and report their achieved offset;
the deployment warning is updated to say so.
Campaigns are armed on deploy, re-armed after each episode finalizes, and
reconciled at agent startup for campaigns deployed in a previous lifetime.
Tests
CC=/usr/bin/clang go test -race ./go/internal/agent/data/... ./go/internal/agent/services/...is green, along with
go build ./go/..., emptygofmt -l go/, and cleango vet. New unit tests:of pictures;
real broadcast sample ids, counted as captured payload;
trigger;
explicit-parameter holder (it never forces the takeover), plus an end-to-end
arm/broadcast/trigger flush on a bare hub;
at the trigger, while a ROS 2 source still does;
max_resolutionand forbuffer plus
rate, and does not fire for a buffer alone, for explicitparameters alone, or for an unarmed snapshot-mode source.
Hardware verification plan
No device installs were performed. On a device:
buffer: 10sand a camera source carrying no explicitmax_resolutionorrate, and confirm deployment no longer warns that thecamera starts at the trigger and does not print the mutual-exclusivity
warning. Deploy a second campaign that sets both a buffer and an explicit
max_resolutionand confirm the mutual-exclusivity warning does print.index.jsonl.canonical_episode_nanosis negative(BEFORE the trigger instant), each carrying a real
sample_id, and thattheir
byte_offset/segmentresolve to decodable payload in the segmentfiles. Join those sample ids against the model-input ledger to confirm frames
a model consumed before the trigger resolve to retained payload.
below
-10sonce armed for at least the buffer, and a shorter (honest)achieved offset when triggered shortly after arming.
Do not squash merge. Do not merge without review.