Skip to content

get-started.md

Latest commit

 

History

History
268 lines (177 loc) · 9.43 KB

get-started.md

File metadata and controls

268 lines (177 loc) · 9.43 KB

Get Started

Live Video Search is a Metro AI Suite sample that adapts the VSS pipeline for semantic search on live Frigate streams. It ingests live camera streams, indexes video segments with embeddings and timestamped camera metadata, and lets users select cameras, time ranges, and free‑text queries to retrieve ranked, playable clips with confidence scores while surfacing live system metrics. This guide starts the Live Video Search stack (Smart NVR + VSS Search) using Docker Compose.

Prerequisites

Project Structure

live-video-search/
├── config/                        # Local configuration and assets
│   ├── frigate-config/            # Frigate camera configs (active + templates)
│   ├── mqtt-config/               # Mosquitto configuration
│   ├── telemetry/                 # Telemetry collector configs
│   └── nginx.conf                 # NGINX reverse proxy
├── data/                           # Runtime data (recordings, caches)
├── docker/                         # Compose files
│   ├── compose.search.yaml        # VSS Search stack
│   ├── compose.smart-nvr.yaml      # Smart NVR stack
│   └── compose.telemetry.yaml      # Telemetry collector
├── docs/                           # Documentation
│   └── user-guide/                # User guides
├── setup.sh                        # Main setup script
└── README.md                       # Project overview

Set Required Environment Variables

Before running the application, you need to set several environment variables:

  1. Configure the registry: The application uses registry URL and tag to pull the required images.

    export REGISTRY_URL=intel
export TAG=latest

  2. Set required credentials for some services: Following variables MUST be set on your current shell before running the setup script:

    # MinIO credentials (object storage)
export MINIO_ROOT_USER=<minio-user>
export MINIO_ROOT_PASSWORD=<minio-pass>

# PostgreSQL credentials (database)
export POSTGRES_USER=<postgres-user>
export POSTGRES_PASSWORD=<postgres-pass>

# Embedding model for search
export EMBEDDING_MODEL_NAME="CLIP/clip-vit-b-32"

# MQTT credentials (Smart NVR)
export MQTT_USER=<mqtt-user>
export MQTT_PASSWORD=<mqtt-pass>

Optional Environment Variables

You can customize the application behavior by setting the following optional environment variables before running the setup script:

  1. Control the frame extraction interval (Video Search Mode):

    The DataPrep microservice samples frames from uploaded videos according to the FRAME_INTERVAL environment variable. Set this variable before running source setup.sh --search to control how often frames are selected for processing.

    export FRAME_INTERVAL=15

    In the example above, DataPrep processes every fifteenth frame: each selected frame (optionally after object detection) is converted into embeddings and stored in the vector database. Lower values improve recall at the cost of higher compute and storage usage, while higher values reduce processing load but may skip important frames. If you do not set this variable, the service falls back to its configured default.

  2. Enable ROI consolidation (Video Search Mode):

    ROI consolidation groups overlapping object detections into merged regions of interest (ROIs) before cropping for embeddings. Enable this feature and tune it with the following environment variables:

    # Enable ROI consolidation (default: false)
export ROI_CONSOLIDATION_ENABLED=true

# IoU threshold for grouping ROIs (higher = stricter merging)
export ROI_CONSOLIDATION_IOU_THRESHOLD=0.2

# Only merge ROIs with the same class label when true
export ROI_CONSOLIDATION_CLASS_AWARE=false

# Expand merged ROIs by a fraction of width/height
export ROI_CONSOLIDATION_CONTEXT_SCALE=0.2

    The IoU calculation follows the standard formula:

    $$ IoU(A, B) = \frac{|A \cap B|}{|A \cup B|} $$

  3. To use GPU acceleration for embedding generation, set the following variable before running the setup script:

    # Enable GPU embeddings
export ENABLE_EMBEDDING_GPU=true

  4. To explicitly select devices for DataPrep and the Multimodal Embedding service, set:

    # CPU or GPU
export VDMS_DATAPREP_DEVICE=GPU
export EMBEDDING_DEVICE=GPU

    If unset, both default to CPU. Setting ENABLE_EMBEDDING_GPU=true forces both to GPU.

Configure Cameras

Edit config/frigate-config/config.yml to add or update camera inputs. This is the active Frigate configuration used at startup.

For reference, see the default template in config/frigate-config/config-default.yml.

Start the Application

source setup.sh --start

RTSP Test Stream (Out-of-Box)

Use the bundled sample video to spin up a looped RTSP stream and point Frigate at it.

  1. Start the stack with the RTSP test services:

    source setup.sh --start-rtsp-test

  2. Confirm the sample stream is live in Frigate:

    • Open http://<host-ip>:5000 and select the rtsp-garage camera.

This uses config/frigate-config/config-rtsp.yml and publishes config/videos/garage.mp4 over RTSP via mediamtx. Replace the RTSP URL in config/frigate-config/config.yml with your real camera streams when moving to production.

Access:

  • VSS UI: http://<host-ip>:12345

USB Camera (Direct Frigate Input)

Use a local USB camera (UVC/V4L2) as the Frigate input without creating an RTSP stream.

  1. Plug in your USB camera and confirm the device node exists (typically /dev/video0).

  2. Start the stack with the USB camera override:

    source setup.sh --start-usb-camera

  3. Open Frigate UI at http://<host-ip>:5000 and select the usb-camera feed.

Notes:

  • If your camera is not /dev/video0, update config/frigate-config/config-usb.yml and/or set USB_CAMERA_DEVICE before starting:

    export USB_CAMERA_DEVICE=/dev/video2
source setup.sh --start-usb-camera

  • You can tune resolution and frame rate in config/frigate-config/config-usb.yml under input_args.

How to Use Live Video Search

This workflow assumes the stack is running and cameras are configured in Frigate.

Step 1: Add Clips to Search

  1. Open VSS UI at http://<host-ip>:12345.
  2. Click Configure Cameras and enable one or more cameras.
  3. Confirm camera streams are live in Frigate (http://<host-ip>:5000).
  4. Allow the watcher to ingest clips from enabled cameras.

Step 2: Run a Search Query

  1. Open VSS UI at http://<host-ip>:12345.
  2. Select one or more cameras.
  3. Set a time range using either:
    • UI time range picker, or
    • Natural‑language query (examples below).
  4. Enter a query and run search.

Example Queries (Time Range Parsing)

  • person seen in last 5 minutes
  • car near garage in the past hour
  • delivery truck last 30 minutes

Step 3: Review Results

Search results include clip timestamps, confidence scores, and metadata. Use the playback controls to jump to the exact event.

Live Video Search - Review Results

Tips

  • If results are empty, confirm cameras are enabled in Configure Cameras and clips have been ingested.
  • Narrow time ranges improve query latency and relevance.
  • If telemetry is not visible, check that vss-collector is running.

Stop or Reset

# Stop all containers
source setup.sh --down

# Remove volumes, live recordings, and app networks
source setup.sh --clean-data

Telemetry

Telemetry is enabled for Live Video Search and shows live system metrics in the VSS UI when the collector is connected.

Troubleshooting

No clips in search results

  • Confirm cameras are enabled in Configure Cameras in VSS UI.
  • Verify VSS_SEARCH_URL in setup.sh points to the internal endpoint.

Search results empty after changing model

  • If you changed EMBEDDING_MODEL_NAME, clean data and re‑ingest:
    • source setup.sh --clean-data
    • source setup.sh --start

Telemetry information is not being displayed

  • Verify vss-collector is running.
  • Check Pipeline Manager status: /manager/metrics/status.

MQTT connection errors

  • Ensure MQTT_USER and MQTT_PASSWORD are set.
  • Confirm mqtt-broker is healthy: docker ps and docker logs mqtt-broker.

Stream disconnects

  • Check Frigate logs for camera connection errors.
  • Confirm RTSP sources are reachable and credentials are valid.

Docker network label mismatch on startup

If startup fails with an error like network docker_live-video-network was found but has incorrect label, clean up stale networks and restart:

  • source setup.sh --clean-data
  • docker network rm docker_live-video-network live-video-network || true
  • source setup.sh --start

For RTSP test mode, start again with:

  • source setup.sh --start-rtsp-test

References