Skip to content

Camera pre-roll via an Arm lifecycle phase - #1844

Open
martien-wdy wants to merge 2 commits into
fix/camera-hub-param-downgradefrom
feat/camera-preroll-arm
Open

Camera pre-roll via an Arm lifecycle phase#1844
martien-wdy wants to merge 2 commits into
fix/camera-hub-param-downgradefrom
feat/camera-preroll-arm

Conversation

@martien-wdy

@martien-wdy martien-wdy commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Problem

A campaign that sets buffer: 10s promises a clip that runs from ten seconds
before the trigger to after_trigger past it. Today that promise holds only for
application records: the data manager keeps a continuously maintained pre-roll
ring, but it walks storedApplicationRecord only. Camera capture begins AT the
trigger, so the frame that caused the episode is never recorded. In the demo the
subject was already in frame rather than walking in. The design's Arm lifecycle
phase for exactly this was never built and is recorded in the debt register as
"No fragment pre-roll (Arm lifecycle unimplemented)".

Cause

The camera capture adapter had no standby state. It joined the producer hub and
opened its episode files (segments, index.jsonl, clock_samples.jsonl) only
when the campaign triggered, so there was no keyframe-aligned buffer of the
seconds before the trigger to flush into the episode.

Solution

Add an Arm (standby) phase to the capture-adapter contract, camera only this
round.

  • Where the phase lives. dataArmingAdapter in
    go/internal/agent/services/data_service.go is the optional pre-roll
    extension of the existing capture-adapter contract (Arm/Disarm). The
    camera adapter implements it; audio and ROS 2 do not. The episode's campaign
    key is threaded to the adapter through a new
    data.CaptureSession.CampaignKey, so the adapter's own Start finds and
    consumes the ring the campaign armed.

  • Arming is non-owning. A parameter-less StreamVideoRequest joins the
    device hub exactly like the sensor path, so arming never takes a running
    stream from a viewer and registers no explicit-parameter holder. That is the
    bit the parameter-precedence work this branch stacks on pivots the takeover
    on, so arming can never force the takeover, and if an unrelated
    explicit-parameter capture restarts the producer, the armed subscriber
    reattaches (like the sensor path) and marks a segment boundary in the ring.

  • The ring reuses existing mechanisms. cameraPreRollRing
    (data_camera_preroll.go) mirrors the application ring's byte-eviction
    discipline (preRollBytes/evictPreRoll): drop the oldest until within the
    window and the byte cap. It is specialised so it only ever begins on a
    random-access unit and only evicts whole groups of pictures, reusing the
    adapter's frameRandomAccess keyframe logic, because a flushed clip that does
    not start on a keyframe will not decode. One ring per campaign source, bounded
    by buffer seconds and by a fixed byte ceiling, whichever is tighter.

  • Real capture times and identities. On trigger the ring is flushed into the
    opening frames on the same subscription, which then continues live. Each
    buffered frame carries the bracketed CLOCK_BOOTTIME receipt the hub stamped
    at broadcast (receiptBootNanos) and its real sample_id, so index.jsonl
    gets true negative-offset canonical_episode_nanos and payload-retention
    accounting and the sample-id/model-input correlation treat them as captured
    frames with their real ids and offsets.

  • Pre-roll and explicit stream parameters are mutually exclusive, and
    deployment says so.
    Because arming asserts no parameters, a camera source
    that sets both a buffer and an explicit max_resolution or rate records
    both its pre-roll and its live tail at the producer's running parameters. The
    manifest reports those values as requested but not achieved, and
    wendy data campaign deploy now warns at deploy time, naming each offending
    source, so the operator learns while they can still change the plan rather
    than after an episode was recorded at the wrong parameters. Snapshot-mode
    camera sources are never armed, so their explicit parameters are still
    honored and they are not warned about.

  • Achieved versus requested, honestly. The manifest source entry keeps the
    requested offset (-buffer) and reports the achieved offset from how far back
    the ring actually reached: at steady state at least the buffer, but shorter
    right after arming. If the ring cannot reach a keyframe at or before
    X - buffer, it flushes from the earliest keyframe it has and records
    achieved < requested. It never fabricates frames and never fails the episode.
    Audio and ROS 2 still start at the trigger and report their achieved offset;
    the deployment warning is updated to say so.

Campaigns are armed on deploy, re-armed after each episode finalizes, and
reconciled at agent startup for campaigns deployed in a previous lifetime.

Tests

CC=/usr/bin/clang go test -race ./go/internal/agent/data/... ./go/internal/agent/services/...
is green, along with go build ./go/..., empty gofmt -l go/, and clean
go vet. New unit tests:

  • the armed ring retains keyframe-aligned frames and evicts whole older groups
    of pictures;
  • a trigger flushes the pre-roll with correct negative canonical offsets and the
    real broadcast sample ids, counted as captured payload;
  • achieved < requested is reported when armed less than a buffer before the
    trigger;
  • arming joins a defaulted hub without restarting it and without registering an
    explicit-parameter holder (it never forces the takeover), plus an end-to-end
    arm/broadcast/trigger flush on a bare hub;
  • a buffered camera-only campaign no longer deploy-warns that its stream starts
    at the trigger, while a ROS 2 source still does;
  • the mutual-exclusivity warning fires for buffer plus max_resolution and for
    buffer plus rate, and does not fire for a buffer alone, for explicit
    parameters alone, or for an unarmed snapshot-mode source.

Hardware verification plan

No device installs were performed. On a device:

  1. Deploy a campaign with buffer: 10s and a camera source carrying no explicit
    max_resolution or rate, and confirm deployment no longer warns that the
    camera starts at the trigger and does not print the mutual-exclusivity
    warning. Deploy a second campaign that sets both a buffer and an explicit
    max_resolution and confirm the mutual-exclusivity warning does print.
  2. Wait past the buffer, then walk into frame to fire the trigger.
  3. Download the sealed clip and open the camera source's index.jsonl.
  4. Confirm index entries exist whose canonical_episode_nanos is negative
    (BEFORE the trigger instant), each carrying a real sample_id, and that
    their byte_offset/segment resolve to decodable payload in the segment
    files. Join those sample ids against the model-input ledger to confirm frames
    a model consumed before the trigger resolve to retained payload.
  5. Confirm the manifest's camera source entry reports an achieved offset at or
    below -10s once armed for at least the buffer, and a shorter (honest)
    achieved offset when triggered shortly after arming.

Do not squash merge. Do not merge without review.

Give camera capture a pre-roll so a triggered episode's video starts BEFORE
the trigger, making a campaign's `buffer` mean what it says for the camera and
not only for application records.

Problem: today the data manager keeps a continuously maintained pre-roll ring,
but it holds only application records; camera capture begins AT the trigger, so
the frame that caused the episode is never recorded. The design's Arm lifecycle
phase was never built (the debt register lists "No fragment pre-roll").

Cause: the camera capture adapter had no standby state. It subscribed to the
producer hub and opened its episode files only when the campaign triggered, so
there was no keyframe-aligned buffer of the seconds before the trigger to flush.

Solution: add an Arm (standby) phase to the capture-adapter contract. A campaign
that requests a buffer arms its continuous-mode camera sources: each subscribes
to the device hub as a NON-owning consumer (an empty stream request, exactly
like the sensor path, so it never takes a stream from a viewer and never
registers an explicit-parameter holder that would force the parameter-precedence
takeover from the hub-state work this stacks on) and keeps a keyframe-aligned
encoded ring of `buffer` seconds. On trigger the ring is flushed into the
opening camera frames at their real negative-offset canonical times, on the same
subscription that then continues live. The ring reuses the application ring's
byte-eviction discipline and the adapter's existing keyframe logic
(frameRandomAccess), and only ever evicts whole groups of pictures so a flushed
clip decodes. Frames carry the bracketed CLOCK_BOOTTIME receipt the hub stamped
at broadcast and their real sample identities, so payload-retention accounting
and the sample-id/model-input correlation treat them as captured frames. The
manifest reports requested versus achieved pre-roll honestly: achieved is at
least the buffer at steady state but shorter when armed too recently, and the
episode is never failed and frames are never fabricated. Audio and ROS 2 still
start at the trigger this round and report their achieved offset honestly.
…ters

Pre-roll arms a standby subscription that asserts no stream parameters, so a
camera source that sets BOTH a buffer and an explicit max_resolution or rate
records at the producer's running parameters instead. The episode manifest
already reported this honestly, but only after the fact: the operator learned
about it by reading the manifest of an episode already recorded at the wrong
parameters. That is the same class of silent downgrade the parameter-precedence
work this stacks on exists to prevent, arriving through a different path.

Warn at deployment, where the operator can still change the plan, following the
existing deploy-warning pattern (unimplemented capture modes,
retention.local_quota). The warning names each offending source and its explicit
values, states that pre-roll and explicit stream parameters are mutually
exclusive in this release, and says the values will be reported as requested but
not achieved.

Only sources that are actually armed conflict. Snapshot-mode camera sources are
never armed, so their explicit parameters are still honored through the normal
capture join and they are deliberately not named. The runtime behavior is
unchanged: arming must not assert parameters, both because it would take a
running camera from a viewer at deploy time for a campaign that may never
trigger, and because asserting them at the trigger would change the sequence
parameter set part way through a clip and cost the episode its browser-playable
rendering.

The campaign documentation for capture.buffer now states the mutual exclusivity.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant