Skip to content

claude.md

Latest commit

 

History

History
211 lines (159 loc) · 6.49 KB

claude.md

File metadata and controls

211 lines (159 loc) · 6.49 KB
Raw

ReVision-RT

Real-time motion-adaptive seam carving system for content-aware video resizing.

Project Overview

This is a C++17 project that implements real-time seam carving with motion awareness. It processes video streams (camera or file) and dynamically removes vertical seams to reduce image width while preserving important content and respecting motion.

Build System

  • CMake (minimum 3.16)
  • C++17 required
  • Dependencies: OpenCV, ONNX Runtime, pthreads
mkdir build && cd build
cmake ..
make

Key Executables

Target Description
seamCarvingApp Main application (live camera)
cameraTest Camera test with custom width ratio
videoTest Video file processing
batchProcessor Batch video processing
depthTest Depth estimation testing
fusionTest Energy fusion testing

Architecture

Threading Model

The system uses a multi-threaded architecture with specialized threads:

Thread File Purpose
CarverThread src/threads/carver.cpp Seam removal and validation
MotionThread src/threads/motion.cpp Motion detection and importance updates
DisplayThread src/threads/display.cpp Real-time display output
RetargetThread src/threads/retarget.cpp Frame retargeting

All threads inherit from BaseThread (src/threads/base.hpp) which provides start/stop lifecycle management.

Core Data Structures

SharedData (src/core/data.hpp):

  • Central data container shared across all threads
  • Contains cv::Mat frames (input, gray, packed, motion)
  • Manages pixel grid (PixelCore) and validity maps
  • Thread-safe with separate mutexes for data vs structure
  • Validation queue for pixel revalidation

PixelCore (src/core/pixelCore.hpp):

  • Per-pixel state: importance (base/current), energy, motion values
  • Seam linking (up/down pointers for seam path tracking)
  • Original coordinates for remapping
  • Motion accumulator with residue for temporal smoothing

Algorithm Components

Located in src/algorithms/:

File Purpose
seamCarving.cpp/hpp Core seam carving with forward energy
sobel.cpp/hpp Sobel edge detection for base importance
energyFuser.cpp/hpp Combines Sobel + motion into final importance
motionFusion.cpp/hpp Motion map to pixel importance integration
motionParameters.cpp/hpp Adaptive motion parameters
motionStatistics.cpp/hpp Motion statistics tracking
depthEstimator.cpp/hpp ONNX-based depth estimation
saliencyEstimator.cpp/hpp Saliency detection
u2NetEstimator.cpp/hpp U2-Net based estimation

Seam Selection Strategies (src/algorithms/strategies/seamSelection/):

  • SeamSelectionStrategy - Base interface
  • BasicSeamSelectionStrategy - Standard minimum energy seam
  • RandomSeamSelectionStrategy - Randomized selection

Utilities

Located in src/utils/:

File Purpose
config.cpp/hpp INI file configuration loading
logger.cpp/hpp Logging with levels (DEBUG, INFO, WARN, ERROR)
fpsTracker.cpp/hpp FPS measurement and tracking
parameterManager.cpp/hpp Runtime parameter management
frameProcessor.cpp/hpp Frame preprocessing
appRunner.cpp/hpp Application lifecycle management
visual.cpp/hpp Visualization utilities

Configuration

Configuration is in config.ini:

[frequencies]
display_fps=60.0          # Display refresh rate
motion_fps=30.0           # Motion processing rate
seam_carving_fps=2.0      # Seam removal rate

[algorithm]
decay_min=0.8             # Motion decay minimum
decay_max=0.95            # Motion decay maximum
importance_alpha=0.7      # Sobel weight
importance_beta=0.3       # Motion weight

[debug]
enable_logging=true
log_level=INFO

Key Constants

  • MAX_WIDTH = 800
  • MAX_HEIGHT = 450

Code Patterns

Thread-Safe Data Access

// For cv::Mat access
std::lock_guard<std::mutex> guard(sharedData.getDataMutex());

// For structure/validity access
std::lock_guard<std::mutex> guard(sharedData.getStructureMutex());

Energy Calculation Flow

  1. Sobel computes importanceBase (edge strength)
  2. Motion thread updates motion values in PixelCore
  3. EnergyFuser::fuseEnergy() combines into importanceCurrent
  4. SeamCarving::calculateEnergy() computes forward energy
  5. Seam with minimum cumulative energy is selected and invalidated

Pixel Validation Queue

Pixels flagged for revalidation are queued and processed by the carver:

sharedData.queuePixelForValidation(pixel);
// Later, carver calls:
PixelCore* pixel = sharedData.getNextPixelForValidation();

Directory Structure

src/
├── algorithms/          # Seam carving, energy, motion algorithms
│   └── strategies/      # Selection strategy patterns
├── core/               # Data structures (SharedData, PixelCore)
├── output/             # Display management
├── threads/            # Thread classes (carver, motion, display)
├── utils/              # Config, logging, utilities
├── main.cpp            # Entry point
├── seamCarvingApp.cpp  # Main application class
tests/
├── cameraTest.cpp
├── videoTest.cpp
├── batchProcessor.cpp
├── depthTest.cpp
└── fusionTest.cpp

Common Tasks

Adding a new algorithm

  1. Create src/algorithms/myAlgorithm.cpp and .hpp
  2. Add to CMakeLists.txt under seam_carving_core sources
  3. Include and use in relevant thread or processor

Adding a new thread

  1. Create src/threads/myThread.cpp and .hpp
  2. Inherit from BaseThread
  3. Implement start() and runLoop()
  4. Add to CMakeLists.txt
  5. Instantiate in main application

Modifying energy calculation

  • Base importance: src/algorithms/sobel.cpp
  • Motion contribution: src/algorithms/motionFusion.cpp
  • Fusion: src/algorithms/energyFuser.cpp
  • Forward energy DP: src/algorithms/seamCarving.cpp

Testing

Tests are in tests/ directory. Run from build directory:

./cameraTest 0.7          # Camera with 70% width retention
./videoTest video.mp4 0.5 # Video with 50% width retention
./batchProcessor 0.6 video.mp4

Important Notes

  • The data/ and build/ folders should be ignored
  • OpenCV is used extensively for image processing
  • ONNX Runtime is used for neural network inference (depth, saliency)
  • All threads coordinate through SharedData with mutex protection
  • The carver thread has two states: INITIAL_CARVING and MAINTENANCE