Skip to content

README.md

Latest commit

 

History

History
211 lines (162 loc) · 7.36 KB

README.md

File metadata and controls

211 lines (162 loc) · 7.36 KB
Raw

Supersede

Supersede is a video-understanding app for uploading footage, extracting frames, grouping nearby frames into temporal captions, deriving safety/security events, summarizing each video, and searching events with natural language.

The current implementation keeps the existing React UI and ports the hierarchical pipeline from Video-Understand-Hierarchical into the Express backend. Supabase provides email/password auth, Postgres, pgvector search, and private object storage. OpenAI provides vision captioning, summaries, and embeddings.

Features

  • Natural-language search over indexed footage, backed by pgvector semantic search with a keyword fallback.
  • Visual search results. Each match renders the frame captured closest to the event's midpoint with a short description, relevance score, and time range — reviewable at a glance instead of blank cards.
  • Actionable safety insights. Every high/medium-risk event surfaces concrete "what to do to stay safe" recommendations, derived from detected PPE violations and hazard cues in the summary. They appear on the video detail timeline and on search result cards.
  • Per-footage incident reports. Each analysed video gets a consolidated report — executive summary, risk metrics, a "what went wrong" breakdown with violation tags, and aggregated recommended actions — that managers can download as a paginated PDF.
  • Live indexing status with per-video progress while frames, captions, events, embeddings, and summaries are generated.
  • Instant loads. The React Query cache is tuned and persisted to local storage, so revisiting a page restores data immediately and revalidates in the background instead of re-hitting the API on every refresh.
  • Resilient sessions. Access tokens auto-refresh, and an expired session is recovered transparently (or cleanly routed to login) instead of surfacing errors.

Screenshots

Overview dashboard
Overview — system metrics and recent activity. 		Search results
Search — natural-language results with matched frames, relevance scores, and safety insights.
Incident report
Incident Report — per-footage summary, risk metrics, and recommended actions (exportable to PDF). 		Secure access
Secure access — Supabase email/password authentication.

Screenshots use representative sample data.

Architecture

  1. The React app authenticates users with Supabase email/password auth.
  2. The API validates each bearer token with Supabase Auth.
  3. Videos upload directly to Supabase Storage using signed upload URLs.
  4. The backend downloads each uploaded video, runs ffprobe/ffmpeg, and stores extracted frames in Supabase Storage.
  5. OpenAI vision models caption overlapping frame groups.
  6. The aggregator turns group captions into events, violations, and per-event risk levels.
  7. OpenAI embeddings are stored in Postgres vector(1536) columns for semantic search.
  8. The summarizer stores a video-level summary, important events, activities, and risk level.

Tech Stack

Layer Technology
Frontend React 19, Vite, Tailwind CSS, shadcn-style components
Backend Express 5, TypeScript
Auth Supabase Auth, email/password
Database Supabase Postgres, Drizzle ORM, pgvector
Storage Supabase Storage private buckets
AI OpenAI SDK
Video Processing ffmpeg, ffprobe
Monorepo pnpm workspaces

Project Structure

artifacts/
  api-server/       Express backend
  cctv-search/      React frontend
  mockup-sandbox/   Component sandbox
lib/
  api-spec/         OpenAPI spec
  api-zod/          Generated Zod validators
  api-client-react/ Generated React Query client
  db/               Drizzle schema
  integrations-openai-ai-server/
supabase/
  migrations/       Supabase SQL migrations

Local Setup

Install dependencies:

pnpm install

Create a Supabase project, then create private Storage buckets named videos and frames.

Apply the migration in supabase/migrations/20260627000000_video_understanding_schema.sql through the Supabase SQL editor or the Supabase CLI.

Copy the example environment file and fill in your own values:

cp .env.example .env

Start the API:

PORT=3000 pnpm --filter @workspace/api-server run dev

Start the frontend:

pnpm --filter @workspace/cctv-search run dev

Environment

Server variables:

PORT=3000
DATABASE_URL=postgresql://...
SUPABASE_URL=https://your-project.supabase.co
SUPABASE_SERVICE_ROLE_KEY=...
SUPABASE_STORAGE_BUCKET_VIDEOS=videos
SUPABASE_STORAGE_BUCKET_FRAMES=frames
OPENAI_API_KEY=...
OPENAI_VISION_MODEL=gpt-5-mini
OPENAI_SUMMARY_MODEL=gpt-5-mini
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_EMBEDDING_DIMS=1536
FRAME_SAMPLE_FPS=2
CAPTION_GROUP_SIZE=4
CAPTION_GROUP_STRIDE=2

Frontend variables:

VITE_API_BASE_URL=http://localhost:3000
VITE_SUPABASE_URL=https://your-project.supabase.co
VITE_SUPABASE_ANON_KEY=...

Do not commit real API keys or Supabase service-role keys.

Hosting

Recommended split:

  • Supabase hosts Postgres, Auth, pgvector, and private Storage.
  • Vercel or Netlify can host the Vite frontend as static assets.
  • Render, Fly.io, Railway, or another long-running Node host should run the Express API because it invokes ffmpeg and may process videos longer than typical serverless timeouts.

Set the frontend VITE_API_BASE_URL to the deployed API origin. Set API CORS to allow the frontend origin before production launch.

For production, configure Supabase Auth redirect URLs for the deployed frontend, keep the service-role key only on the backend host, and ensure ffmpeg/ffprobe are available in the backend runtime image.

Frontend deploy settings for Vercel are in vercel.json:

Install Command: pnpm install --frozen-lockfile
Build Command: pnpm --filter @workspace/cctv-search run build
Output Directory: artifacts/cctv-search/dist/public

Backend deploy settings for a Docker-based web service:

Dockerfile: artifacts/api-server/Dockerfile
Docker context: .
Start command: handled by Docker CMD

Backend production environment variables should include PORT, DATABASE_URL, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, Storage bucket names, and OPENAI_API_KEY.

Scripts

pnpm run typecheck
pnpm run build
pnpm --filter @workspace/api-spec run codegen

API Surface

Main endpoints:

  • POST /api/storage/uploads/request-url
  • GET /api/videos
  • POST /api/videos
  • GET /api/videos/:id
  • POST /api/videos/:id/index
  • GET /api/videos/:id/index-status
  • GET /api/videos/:id/frames
  • GET /api/videos/:id/events
  • GET /api/videos/:id/summary
  • POST /api/search
  • GET /api/search/history

Database

The Supabase migration creates these user-scoped tables:

  • videos
  • frames
  • frame_groups
  • events
  • video_summaries
  • search_history

All tables enable row-level security. The API also scopes queries by the authenticated Supabase user id.