Skip to content

DESIGN.md

Latest commit

 

History

History
672 lines (583 loc) · 34.3 KB

DESIGN.md

File metadata and controls

672 lines (583 loc) · 34.3 KB
Raw

Wildlife Pipeline — Design

A multi-worker pipeline that ingests trail-cam clips, filters them for actual wildlife, cleans up the audio, identifies species, and pushes a continuous live stream to YouTube/Twitch.

Target host: Linux GPU server, RTX 4090 (24 GB VRAM), user-level install at ~/wildlife-pipeline. The 10 workers (group wildlife) and the dashboard (standalone dashboard program) all run under supervisord as user bryan — see deploy/supervisor/. (The dashboard previously ran under Docker Compose; some §6 references to the "dashboard container" are now historical.)

1. Goals

  1. Drop raw clips into a per-area inbox and forget about them. An inbox is a stream/area (one folder per area); several physical cameras may feed one.
  2. Aggressively filter out any segment that has no animal in view.
  3. Suppress unwanted audio (humans, dogs, aircraft, vehicles) without creating jarring silences.
  4. Produce a closed-caption sidecar identifying bird and animal species.
  5. Continuously stream the processed output to an RTMP endpoint, with a standby loop when the queue is empty so the stream never drops.
  6. Support multiple cameras funneling into one merged stream — clips play in arrival order (FIFO), prioritizing continuous airtime over chronological ordering, with no on-screen camera label.
  7. Automatically clip highlights (rare/allowlisted/first-seen species or long encounters) and produce a 9:16 vertical variant for social.
  8. Support an offline file-render mode alongside or instead of RTMP: wildlife render-file --duration 1h --output ... produces a single mp4 compilation from processed clips, then exits. Runs in parallel with the live stream if both are wanted.

2. Non-goals (for v1)

  • Horizontal scaling across multiple machines.
  • A web UI / dashboard (CLI + logs only).
  • Re-encoding for multiple bitrate ladders (single 1080p30 output).
  • Cloud storage (everything is local disk).

3. Architecture

Workers are independent processes that communicate by moving files between state directories. Each worker watches its in/ dir via inotify, writes artifacts to work/<clip-id>/, and on success moves the clip into the next stage's in/ dir. This makes restarts safe and pipeline state visible with plain ls.

inbox/
  ├─ wcc/                           ← per-area/stream drop zones (rsync, SD, etc.)
  ├─ cam02/
  └─ ...
       └─ 01_gating/in/             ← ingest watcher tags clip-id with cam-id, routes here
            └─ 02_audio/in/         ← clips with animals + tail buffer
                 └─ 03a_species_visual/in/   ┐ fan-out: visual + audio
                 └─ 03b_species_audio/in/    ┘ ID run in parallel
                      └─ 04_captions/in/     ← assembler waits for both 03 outputs
                           ├─ 05_ready/      ← captioned clips queued for streamer
                           │    └─ archive/  ← compiler moves clips here post-stream
                           └─ 04b_highlights/in/   ← parallel: highlight curator
                                └─ highlights/<date>/<species>_<clip-id>/
failed/<stage>/                     ← any worker that errors moves clip here
state/sightings.db                  ← SQLite: species → last-seen timestamp

The bulky media dirs (inbox/, archive/, highlights/, compilations/, filler_audio/, standby/) live on the 2TB media drive under paths.media_root; pipeline/, failed/, state/, and models/ stay on the fast local disk under paths.root. Clips hop between the two filesystems, so every move uses safe_move() (atomic rename within a fs; copy-then-rename on EXDEV).

After caption-assembler finishes, the clip is hardlinked into both 05_ready/ (for streaming) and 04b_highlights/in/ (for highlight evaluation). The two consumers proceed independently — highlight curation never blocks the stream.

3.1 Worker chain

# Worker Purpose Key tech
I sd-import Watch for removable SD cards; copy videos to inbox, wipe + eject the card udisksctl + /proc/mounts
0 ingest-watcher Watch all per-area inboxes; remux non-mp4 imports (mkv/avi/mov/mts/m2ts) → .mp4; assign <ts>_<area-id> clip-id ffmpeg
1 animal-gate Detect animal frames, cut to animal segments + tail buffer MegaDetector v5 (PyTorch, FP16)
2 audio-cleanup Suppress human/dog/aircraft/vehicle noise; splice donor audio YAMNet (TFLite) + ffmpeg
3a species-visual Crop detections, classify species iNaturalist 2.7k (timm)
3b species-audio Identify bird vocalizations BirdNET-Analyzer
4 caption-assembler Merge 3a + 3b timestamps into .srt; hardlink clip to streamer + highlighter pure Python
4b highlight-curator Detect rare/allowlisted/first-seen species or long encounters → cut + render 9:16 ffmpeg + SQLite
5 stream-compiler FIFO concat of ready clips, burn captions, push RTMP ffmpeg
5b file-compiler One-shot CLI: render N-minute mp4 compilation from clips, then exit ffmpeg
6 idle-filler Loop standby clip when queue empty so stream stays live ffmpeg
— resmon Sample host CPU / memory / GPU / VRAM for the dashboard header nvidia-smi + /proc

03a and 03b run in parallel on the same cleaned clip. The caption-assembler waits on a done marker from both before proceeding. 04b (highlight curator) runs in parallel with 05 (streamer) so highlight extraction never blocks the live stream.

3.2 Why filesystem coordination

  • Debuggable: every stage's state is visible with ls and tail -f.
  • Restart-safe: a worker crash leaves the clip in the previous stage's output dir — nothing is lost.
  • Easy to add stages: drop in a new worker between two dirs.

Tradeoff: doesn't scale across hosts. If that becomes a need, swap inotify for Redis streams without changing worker logic.

3.3 Operations: dashboard & control plane

A FastAPI + HTMX dashboard (wildlife dashboard, containerized) is the operator surface. It reads pipeline state straight off the filesystem and exposes a few controls:

  • Stats (home page /) — lifetime totals (since the last Fresh Start), aggregated mostly from the gate sidecars so it stays fast: videos processed, seconds of good-quality / shaky / animal / human footage, per-species counts (sightings DB), highlights, compilations (count + length), streams (state/stream/sessions.jsonl), and footage ready to publish.
  • Status (/status) — live queue depths per area and per stage, failed clips (with retrigger), recent archive (with replay), highlights, and the sd-import worker's last result.
  • Camera (/axis) — embeds the AXIS Q1659 control GUI (exposure / focus / white balance / image / IR-cut + live 1 fps snapshot) as a tab. The camera uses HTTP digest auth, so the dashboard proxies it (/axis/api/{state,set,focus}, /axis/snapshot.jpg) and the browser never sees the camera password. Logic in dashboard/axis.py (ported from tools/axis_aperture_gui.py); connection params from the optional axis: config block, else AXIS_HOST/AXIS_USER/AXIS_PASS env, else defaults. (The dashboard container must be able to reach the camera IP.)
  • Videos (/videos) — browse + play everything (past compilations/streams, ready queue, highlights, rejects, archive, in-flight pipeline); thumbnails open a modal <video> that streams + seeks from the server. A 🎬 Generate compilation button renders a compilation on demand (length / source / captions). Since the dashboard is containerized, it drops state/control/render.request; the host resmon worker runs wildlife render-file and reports progress in render.status.json (running/done/failed banner) — the same control-file bridge used by Restart-workers.
  • Pause / Resume — toggles for each pipeline stage, each area inbox, and the sd-import worker, each backed by a marker file under state/paused/ (<stage>, camera__<id>, sd-import — the camera__ marker prefix is the area id). The relevant worker skips processing while its marker exists, so items accumulate (clips in a stage's in/, files in an area inbox, cards left mounted) and drain on resume. Marker-file based, so it's restart-safe and works from host or container.
  • Resource header — a slim strip showing host CPU / mem / GPU / VRAM, fed by resmon via state/resources.json.
  • Fresh Start (/fresh-start) — two reset modes, each behind a type-Fresh Start-then-yes confirm: ♻ Reset & re-run gathers every clip (archive, in-flight stages, failed, discarded) into a holding pen ingest ignores, wipes the derived artifacts (highlights, compilations, captions, gate sidecars + crops, species, stream sessions, sightings DB), then releases the clips into the inbox so the whole pipeline reprocesses them (footage kept; holding-pen ordering avoids racing the running workers); ☢ Nuclear deletes all media including source footage. Neither touches code, config, models, or branding.
  • Config editor, How-to, and More info (renders this document).

The container mounts the project at the same absolute path as the host, so the ~/wildlife-pipeline paths in config.yaml resolve identically whether read by a host worker or by the dashboard.

4. Per-worker contracts

Every clip is referenced by a stable clip-id (e.g. 2026-05-28_18-42-07_cam01). A worker's job is: read <stage>/in/<clip-id>.mp4 + any sidecars, write output(s), then move all artifacts to the next stage's in/ dir.

4.1 animal-gate

  • Input: <id>.mp4

  • Output: <id>.mp4 (trimmed), <id>.gate.json (kept ranges, detection log)

  • Logic:

    1. Sample frames at gate.sample_fps (default 4 — dense enough that a visible person can't slip between samples; see person handling below).
    2. Run MegaDetector v5; treat class animal with conf ≥ gate.conf (default 0.35) as a hit.
    3. Merge animal hits into segments, extending each forward by gate.trailing_buffer_s (default 4s) and backward by gate.leading_buffer_s (default 2s). Drop segments backed by fewer than gate.min_detections frames (kills single-frame false positives).
    4. Cut out people (see below).
    5. Cut out shaky sections (see below).
    6. Drop clip entirely if total kept duration < gate.min_keep_s (default 3s).
    7. Re-encode only the kept segments with ffmpeg (select filter, NVENC with x264 fallback). A per-frame stability gate (phase correlation) also stops a shaky frame from seeding an animal segment.

  • People are never shown. MegaDetector also labels person. Unlike animals, a person detection (class 2) is recorded on every sampled frame regardless of the stability gate, and the time windows around people are subtracted from the kept segments — those seconds of video are removed entirely, not just muted. This is deliberately aggressive (a hard "no humans in the output" requirement):

    • gate.person_conf (default 0.15) — low threshold, catch faint/edge detections.
    • gate.person_buffer_s (default 4s) — cut this much time around each detection.
    • gate.person_bridge_s (default 6s) — merge person cuts closer than this, so a person flickering in and out of detection is removed as a single block.
    • sample_fps of 4 ensures a visible person is caught on multiple frames.

    <id>.gate.json records person_frames_cut for auditing. (The audio stage separately mutes human/vehicle sound, since a vehicle can be heard before it's seen.)

  • Shaky handheld sections are cut, not the whole clip. Most footage is locked-off (feeder cams), but some is handheld. A cheap pre-pass (no detector) measures per-frame jerk — the frame-to-frame change in the global-motion vector from phase correlation. This is the signal that separates handheld jitter (high jerk) from a steady camera (~0) and from a smooth pan (high motion but low jerk). Frames whose smoothed jerk exceeds gate.shake.max_jerk are marked, padded, and bridged into windows, which are then subtracted from the kept segments — so only the shaky stretches are removed. If nothing stable is left, the clip is discarded ("too shaky") before the detector even runs. Calibrated at sample_fps: 4: steady < 0.008, mild handheld ~0.012, bad handheld > 0.045; default max_jerk: 0.02 cuts 0–4 % of steady clips and 57–73 % of badly handheld ones. Knobs: gate.shake.{enabled, max_jerk, smooth_s, buffer_s, bridge_s}. <id>.gate.json records shaky_s + a quality block (mean_shift, jerk, blur). Blur (Laplacian variance) is measured but not used to gate — it's too scene-dependent.

4.2 audio-cleanup

  • Input: trimmed <id>.mp4
  • Output: <id>.mp4 (clean audio), <id>.audio.json (suppressed ranges, donor sources)
  • Logic:
    1. Run YAMNet on 0.96s windows (its native frame size).
    2. Mark windows where any of these AudioSet labels exceed audio.conf (default 0.4): Speech, Dog, Aircraft, Helicopter, Motor vehicle (road), Car, Truck, Engine.
    3. For each contiguous "bad" range, find a donor:
      • First, scan the same clip for a "clean" range of equal duration (no bad labels, similar RMS level). Prefer the nearest in time.
      • If none, pick a random file from ~/wildlife-pipeline/filler_audio/ and slice a matching duration.
    4. Crossfade the donor over the bad range (50 ms fade in/out) using ffmpeg afade + amix.
  • Config knobs: 
    audio:
  conf: 0.4
  labels: [Speech, Dog, Aircraft, Helicopter, "Motor vehicle (road)", Car, Truck, Engine]
  crossfade_ms: 50
  donor_search_strategy: in_clip_first   # in_clip_first | filler_only | in_clip_only
  filler_dir: ~/wildlife-pipeline/filler_audio

4.3 species-visual

  • Input: the animal crops the gate saved to state/gate_crops/<id>/ (filename {conf}_{t}.jpg, t = original-clip time), plus <id>.gate.json for the kept-segment list.
  • Output: state/species/<id>.visual.json — clip-level majority {species, common_name, score, votes, uncertain} plus a timeline of per-segment {t_start, t_end, species, common_name, score, uncertain} ranges.
  • Logic: Classify each crop with BioCLIP (CustomLabelsClassifier constrained to species.candidates — far more accurate than the open tree of life). Clip-level result is the majority vote; the timeline is each segment's majority mapped onto the trimmed timeline (adjacent same-species merged). Not iNaturalist, and BioCLIP must run in its own process — loading yolov5 (MegaDetector) + open_clip together clashes on torch checkpoint unpickling, which is why the gate saves crops instead of species-visual re-seeking frames.

4.4 species-audio

  • Input: clean <id>.mp4 (audio only)
  • Output: <id>.species_audio.json — list of {t_start, t_end, species, confidence}
  • Logic: Run BirdNET-Analyzer on the audio track at default 3s windows. Keep detections above species.audio_conf (default 0.5).

4.5 caption-assembler

  • Input: <id>.species_visual.json, <id>.species_audio.json
  • Output: <id>.srt (also writes <id>.vtt for HLS compatibility)
  • Logic (current): timed, per-segment captions. species-visual labels each kept segment (majority vote of that segment's crops) and emits a timeline of {t_start, t_end, species} ranges in the trimmed clip's timeline; adjacent same-species ranges are merged. The assembler writes one .srt/.vtt cue per range, so the on-screen name changes as different animals appear (e.g. Eastern Fox Squirrel → Carolina Chickadee → White-winged Dove). Bird calls (♪ …) from the audio stage are clip-level and appended to every cue. If there's no timeline (e.g. a re-fed clip with no crops) it falls back to a single full-duration caption. species-visual still also writes the clip-level majority species that the highlight curator + sightings DB use.

4.6 highlight-curator

  • Input: a fan-out copy of <id>.mp4 + state/species/<id>.visual.json (with its per-segment timeline).

  • Output: highlights/<YYYY-MM-DD>/<species>_<id>[_<start>s]/ containing:

    • <id>.mp4 — the clipped highlight segment
    • <id>_vertical.mp4 — optional 9:16 crop centered on the subject (subject_cx/cy from the gate sidecar)
    • thumb.jpg — frame from the middle of the segment
    • metadata.json — species, trigger reasons, start/end, source clip-id

  • Logic — walks the species timeline; a range is highlight-worthy if:

    1. Allowlist: its species ∈ highlights.allowlist, OR
    2. First-seen: its species not in state/sightings.db within the last highlights.first_seen_window_days (default 14).

    Each qualifying range is clipped out on its own (± lead_s/tail_s, default 5s/10s), so a brief notable animal inside a longer clip (e.g. a skunk among squirrels) gets its own highlight — the old clip-level logic missed those. Overlapping same-species ranges are merged.

  • State: every distinct species in the clip is recorded in state/sightings.db (keeps the first-seen window accurate).

  • (The earlier long-encounter and rarity rules were dropped: long-encounter fired on ~everything, and rarity was never implemented.)

4.7 stream-compiler

  • Input: clips in 05_ready/, polled by mtime (arrival order)
  • Output: RTMP stream to YouTube/Twitch
  • Logic: Persistent ffmpeg reading a named-pipe concat; a Python controller appends each ready clip (normalized, captions burned via subtitles) in FIFO arrival order — goal is continuous airtime, not real-world timeline. Each clip gets a fade in/out (stream.crossfade_s, default 0.5s) so transitions are smooth rather than hard cuts. (A true cross-dissolve between clips needs a look-ahead buffer — the file-compiler does that on static renders; a real-time concat stream can't without rearchitecting, so the live path uses a fade.) When the queue is empty it loops standby/loop.mp4 so the stream never drops. Streamed clips move to archive/<YYYY-MM-DD>/, and each session is appended to state/stream/sessions.jsonl for the Stats page. Until a real rtmp_url is set it runs in archive mode (local files only, no push).

4.7b file-compiler (CLI: wildlife render-file)

  • Invocation: one-shot CLI, not a long-running service. Runs whenever you want a compilation (manually, on a cron, etc). 
    wildlife render-file --duration 1h --output ~/comp/today.mp4 \
                      [--source archive|05_ready] \
                      [--since 2026-05-29] [--until 2026-05-29] \
                      [--no-captions]
  • Logic:
    1. Gather candidate clips from --source (default archive/, newest-first by mtime, optionally filtered by --since/--until).
    2. Greedily select clips in arrival order until total duration ≥ --duration, normalize each to a uniform format, then crossfade between them — xfade (video) + acrossfade (audio), file_output.crossfade_s (default 0.75s), clamped to half the shortest clip — so they dissolve instead of hard-cutting. NVENC re-encode at config resolution/bitrate; falls back to a hard-cut concat when crossfade is 0 or there's a single clip. Every clip is given a silent track if it lacks one (so acrossfade always has audio).
    3. Burn captions from each clip's .srt sidecar (matching stem) unless --no-captions.
    4. Write to --output (or file_output.default_output_dir/comp_<ts>.mp4 if omitted), then exit.
    5. Move the used clips out of the pool (default; --move-used/--keep-used, or file_output.move_used) into file_output.used_dir (default <root>/used, date subfolder preserved) — so the next compilation draws from what's left instead of re-selecting the same clips (otherwise every compilation is identical). Only clips that made it into the output are moved, and only after the output is written. The dashboard's Generate-compilation modal exposes this as a checkbox, and the Videos page lists moved clips under Used in compilations.
  • Coexists with streaming: completely independent process. The two read from different sources (archive/ vs 05_ready/) and never contend. If you want a file built from clips that haven't streamed yet, pass --source 05_ready.
  • Defaults from config: file_output.default_duration_s, file_output.default_output_dir, file_output.resolution, file_output.fps, file_output.bitrate_kbps, file_output.caption_overlay.

4.8 idle-filler

  • Loops ~/wildlife-pipeline/standby/loop.mp4 into the same RTMP target using a hot-swap with stream-compiler. Implemented as a second ffmpeg with a tee/select multiplexer, OR (simpler v1) the compiler itself inserts the standby loop into its concat list when no clips are ready. v1 choice: keep it inside stream-compiler to avoid the hot-swap complexity. idle-filler is reserved for v2 if needed.

4.9 sd-import

  • Polls /proc/mounts for mountpoints appearing directly under sd_import.mount_root (where udisks2 auto-mounts removable media).
  • For each newly mounted, eligible card it: copies every video (sd_import.video_exts) into the import directory as a verified copy (.part → size-check → rename), deletes the originals, then udisksctl unmount + power-off (eject). Destination is sd_import.import_dir (absolute, or relative to inbox/); if empty, the first area's inbox folder. Progress is written live (current.imported/total) and cumulative total_imported/total_cards survive worker restarts.
  • Eligibility (all required): mountpoint directly under mount_root, name not in sd_import.ignore, and the backing device is removable (/sys/block/<dev>/removable, with mmcblk* always treated as removable). Internal disks are therefore never touched even if the ignore list is empty. Note: the 2TB media drive mounts under mount_root (/media/bryan/2TB), so it is listed in sd_import.ignore and is non-removable — doubly excluded.
  • Originals are deleted only after a verified copy; a card is ejected only if every file imported with no error. Writes state/sd_import.json (current status + recent events) for the dashboard.
  • Config: sd_import.{enabled,mount_root,ignore,require_removable,import_dir, video_exts,delete_after,eject_after,poll_interval_s}.
  • The ingest watcher accepts the same video_exts (plus .mp4); the pipeline is .mp4-only, so a non-mp4 import (mkv/avi/mov/mts/m2ts) is remuxed to .mp4 (ffmpeg -c copy -f mp4, lossless) on ingest. If the codecs aren't mp4-muxable it's moved under an .mp4 name and the gate re-encodes it to a true mp4.

4.10 resmon

  • Host resource monitor. Samples CPU% (/proc/stat delta), memory% (/proc/meminfo), and GPU util + VRAM% (nvidia-smi), writing state/resources.json every resmon.interval_s for the dashboard header.
  • Runs on the host (the dashboard container has no GPU access); GPU fields are null when nvidia-smi is unavailable.

5. Config

Single ~/wildlife-pipeline/config.yaml loaded by every worker. Workers re-read on SIGHUP (no restart needed for tuning).

Paths split across two filesystems. Working/scratch dirs (pipeline/, failed/, state/, models/) stay under root on the fast local disk; the bulky media dirs live under media_root on the 2TB drive. Because clips therefore hop between filesystems, all moves go through safe_move(), which falls back to copy-then-atomic-rename on EXDEV instead of failing.

paths:
  root: ~/wildlife-pipeline                          # local disk: pipeline scratch, state, models
  media_root: /media/bryan/2TB/wildlife-pipeline     # 2TB drive: all bulky media
  inbox: /media/bryan/2TB/wildlife-pipeline/inbox    # per-area/stream subdirs inside
  filler_audio: /media/bryan/2TB/wildlife-pipeline/filler_audio
  standby: /media/bryan/2TB/wildlife-pipeline/standby/loop.mp4
  state_db: ~/wildlife-pipeline/state/sightings.db   # stays local (frequent writes)
  highlights: /media/bryan/2TB/wildlife-pipeline/highlights
  compilations: /media/bryan/2TB/wildlife-pipeline/compilations
  archive: /media/bryan/2TB/wildlife-pipeline/archive

cameras:                     # each entry is an AREA / stream (one or more physical cameras feed it)
  - id: wcc
    name: "Backyard Feeder"
    # inbox path defaults to {paths.inbox}/{id}
  - id: cam02
    name: "Trail near pond"

gate:
  sample_fps: 4              # frames/sec scanned (dense so people can't slip between samples)
  conf: 0.35                 # animal min confidence
  trailing_buffer_s: 4
  leading_buffer_s: 2
  min_keep_s: 3
  min_detections: 2          # a kept segment needs >= this many animal frames
  person_conf: 0.15          # low: detect people aggressively — they are cut from the output
  person_buffer_s: 4         # cut this many seconds around every person detection
  person_bridge_s: 6         # merge person cuts within this gap (flicker = one cut)
  stability:                 # drop shaky/camera-moving frames
    enabled: true
    max_shift_frac: 0.05
    min_sharpness: 0

audio:
  conf: 0.4
  labels: [Speech, Dog, Aircraft, Helicopter, "Motor vehicle (road)", Car, Truck, Engine]
  crossfade_ms: 50
  donor_search_strategy: in_clip_first

species:
  visual_conf: 0.5
  audio_conf: 0.5

highlights:
  enabled: true
  allowlist: [bobcat, "red fox", "barred owl", "pileated woodpecker"]
  first_seen_window_days: 14
  region: "US-TX"            # used for rarity lookup
  rarity_conf: 0.7
  long_encounter_s: 30
  lead_s: 5
  tail_s: 10
  render_vertical: true      # 9:16 crop centered on detection bbox

stream:
  rtmp_url: rtmp://a.rtmp.youtube.com/live2/STREAM-KEY-HERE
  resolution: 1920x1080
  fps: 30
  bitrate_kbps: 6000
  idle_grace_s: 5
  burn_captions: true
  order: arrival             # FIFO by mtime, not chronological
  camera_overlay: false

file_output:
  default_duration_s: 3600           # 1 hour
  default_output_dir: ~/wildlife-pipeline/compilations
  default_source: archive            # archive | 05_ready
  resolution: 1920x1080
  fps: 30
  bitrate_kbps: 6000
  caption_overlay: true

models:
  megadetector: ~/wildlife-pipeline/models/md_v5a.0.0.pt
  inaturalist:  ~/wildlife-pipeline/models/inat_2021_taxa.pt
  yamnet:       ~/wildlife-pipeline/models/yamnet.tflite
  birdnet:      ~/wildlife-pipeline/models/BirdNET_GLOBAL_6K_V2.4.tflite

Secrets (STREAM-KEY) live in ~/wildlife-pipeline/.env, not in the YAML.

6. Directory layout

~/wildlife-pipeline/
├── config.yaml
├── .env                           # RTMP keys, etc.
├── DESIGN.md
├── pyproject.toml                 # uv-managed project
├── src/wildlife/
│   ├── workers/
│   │   ├── ingest.py
│   │   ├── animal_gate.py
│   │   ├── audio_cleanup.py
│   │   ├── species_visual.py
│   │   ├── species_audio.py
│   │   ├── caption_assembler.py
│   │   └── stream_compiler.py
│   ├── common/
│   │   ├── pipeline_fs.py         # stage moves, lockfiles, atomic renames
│   │   ├── ffmpeg_ops.py
│   │   └── config.py
│   └── cli.py                     # `wildlife <worker>` entrypoint
├── systemd/                       # one .service unit per worker
├── models/                        # downloaded weights (gitignored)
├── inbox/
│   ├── wcc/                       # per-area/stream drop zones
│   ├── cam02/
│   └── ...
├── pipeline/
│   ├── 01_gating/   { in/, work/ }
│   ├── 02_audio/    { in/, work/ }
│   ├── 03a_species_visual/ { in/, work/ }
│   ├── 03b_species_audio/  { in/, work/ }
│   ├── 04_captions/ { in/, work/ }
│   ├── 04b_highlights/ { in/, work/ }
│   └── 05_ready/
├── archive/                       # post-stream clips, dated subdirs
├── highlights/<YYYY-MM-DD>/<species>_<clip-id>/  # horizontal.mp4 + vertical.mp4 + thumb + metadata
├── compilations/                  # `wildlife render-file` output mp4s
├── state/sightings.db             # SQLite: species → last-seen timestamp
├── failed/<stage>/                # quarantine for clips that error
├── standby/loop.mp4
└── filler_audio/                  # donor clips for audio splicing

7. Failure handling

  • Atomic moves: workers write to work/<id>/ then mv into the next stage's in/ (same filesystem → atomic rename).
  • Lockfile per clip: pipeline/<stage>/work/<id>/.lock prevents two workers grabbing the same clip after a restart.
  • Quarantine: on exception, the clip + a .error.txt traceback is moved to failed/<stage>/. The worker continues.
  • Idempotency: each worker checks <id>.<stage>.done before processing. Replaying a failed clip is just mv failed/<stage>/<id>* <stage>/in/.
  • Stream resilience: stream-compiler is a single long-lived ffmpeg with -reconnect 1 -reconnect_streamed 1. If it crashes, supervisord restarts it; the standby loop kicks in until ready clips re-flow.

8. GPU / performance budget (RTX 4090)

Rough memory & throughput estimates running concurrently:

Worker VRAM Real-time factor (per clip)
animal-gate (MegaDetector v5, FP16, batch 8) ~3 GB ~5× faster than realtime
species-visual (iNaturalist, FP16) ~2 GB ~10×
species-audio (BirdNET, CPU is fine) 0 ~20×
audio-cleanup (YAMNet, CPU/TFLite) 0 ~30×
stream-compiler (ffmpeg NVENC) ~1 GB realtime (by design)
highlight-curator (ffmpeg NVENC cut + 9:16 crop) ~1 GB ~5× (background, non-blocking)

Total steady-state VRAM well under 8 GB → headroom for batching, future upgrades to larger species models, or running a redundant gate at higher sample_fps for sensitive cameras.

8.1 ffmpeg GPU offload (NVDEC/NVENC)

Originally every ffmpeg decode/encode ran on the CPU (load ~36/32 cores; GPU idle at ~14 W). src/wildlife/common/ffmpeg_ops.py now offloads video to the 4090, with CPU fallback, configured under gpu: in config.yaml. Wired into the gate (_sample_frames, _trim), both compilers, the streamer, and highlights; a per-stage lock in StageWatcher stops a worker decoding two clips at once.

  • Decode → NVDEC via the explicit -c:v <codec>_cuvid decoder, not -hwaccel cuda (which initializes but does not actually offload on this build — measured identical CPU time + 0 % decoder util; cuvid is ~27× cheaper: utime 25.8 s → 0.96 s).
  • Encode → NVENC (h264_nvenc), bounded by a cross-process flock semaphore (gpu.max_encoders) so concurrent sessions stay under the GeForce cap (measured 4 on driver 535.309.01; set to 3 with headroom). Without this the cap was exceeded under load and ffmpeg silently fell back to libx264 (CPU). nvidia-patch (to lift the cap) has no build for this driver yet.
  • 5K width cap → 4K. H.264 NVDEC/NVENC cap out at 4096 px wide, but a large share of trail-cam footage is 5120-wide (5K). ffmpeg_ops.scale_filter downscales clips wider than gpu.max_encode_width (default 3840 = 4K UHD, 0 = off) before the encode, so NVENC engages instead of libx264. Applied at the two source-resolution re-encode sites — the gate trim (the resolution gatekeeper: capping here cascades ≤3840 to every downstream clip, so all later GPU encodes are safe) and the highlight horizontal render; the compilers already normalize to 1080p. NOTE: the 5K decode for frame-sampling still runs on CPU (5120-wide H.264 also exceeds the NVDEC 4096 cap) — only the encode moves to the GPU. Future 5K HEVC footage gets GPU decode (hevc_cuvid, 8192) and the capped GPU encode.
  • Detection → batched. _detections now runs MegaDetector via generate_detections_one_batch (gate.detector_batch, default 8) instead of per-frame — measured ~1.65× (21→35 fps) with identical detections, GPU to 100 % (per-frame left it at ~45 %). Sampled frames are downscaled to 1280 first, so this helps even 5K-source clips.
  • HEVC width: NVDEC H.264 caps at 4096 wide; HEVC/VP9/AV1 reach 8192, so decode_args uses per-codec width limits — 5K HEVC decodes on hevc_cuvid, 5K H.264 stays on CPU.
  • Result: load ~36 → ~4 on GPU-eligible clips (higher while CPU-bound 5K H.264 clips run); GPU decoder 20–100 %, NVENC active, ~80 W under load. Decode, encode, and detection now all use the 4090.

9. Deployment

  • Install: uv sync inside ~/wildlife-pipeline/.
  • Models: scripts/fetch-models.sh pulls MegaDetector / iNaturalist / YAMNet / BirdNET weights into models/.
  • Services: workers run under supervisord — bash deploy/supervisor/install-supervisor.sh to deploy/reload; control with sudo supervisorctl {status,start,stop,restart} wildlife:*.
  • Logs: state/logs/<worker>.log (or sudo supervisorctl tail -f wildlife:animal-gate).
  • Hot config reload: pkill -HUP -f 'wildlife '.

10. Open questions / decisions deferred to v2

Folded into v1: multi-area/stream support (§3, §4.0, §5 cameras — each entry is an area/stream, possibly fed by several physical cameras), highlight clipping with 9:16 vertical render (§4.6, §5 highlights).

Still deferred to v2:

  • A health worker that emits Prometheus metrics.
  • Re-running stages on demand (e.g. re-caption with a better model) — currently requires manual mv.
  • Caption track in multiple languages (post-translate via local LLM).
  • Auto-upload of highlights to YouTube/social (currently just stages files).
  • Per-area tuning overrides (e.g. a creek area in heavy brush gets a lower gate confidence threshold). Trivial to add — just merge a per-area config block at worker startup.

11. Next steps after design approval

  1. Scaffold project skeleton (pyproject.toml, src/wildlife/, empty worker stubs that just move files).
  2. Wire pipeline_fs.py + config.py so the chain works end-to-end with no-op workers (validates plumbing).
  3. Implement workers in order: gate → audio → species-visual → species-audio → captions → stream.
  4. Write a scripts/seed-test-clips.sh that drops a known set of clips into inbox/ for integration testing.