A real-time background replacement application that uses advanced image processing techniques to segment foreground subjects and composite them onto custom backgrounds. The system features a PyQt5-based GUI with live camera feed processing and multiple segmentation pipelines.
- Real-Time Processing: Live camera feed with minimal latency
- Dual Pipeline Architecture:
- Custom segmentation pipeline using background subtraction and guided filtering
- MediaPipe-based segmentation for high-quality results
- Interactive GUI: User-friendly interface built with PyQt5
- Dynamic Background Selection: Choose from preset backgrounds or upload custom images
- Background Initialization: Capture and learn static backgrounds for improved segmentation
- Guided Filter Refinement: Edge-aware mask refinement for natural compositing
- Alpha Blending: Smooth foreground-background composition
The application implements two parallel processing pipelines:
-
Custom Pipeline (
src/pipeline/pipeline.py)- Background modeling using median-based static background capture
- Color-based segmentation in LAB color space
- Guided filter for mask refinement
- Alpha blending for final composition
-
MediaPipe Pipeline (
src/pipeline/ideal_pipeline.py)- MediaPipe Selfie Segmentation for robust person detection
- Guided filter post-processing
- High-quality segmentation masks
- Background Modeling (
background_modeling.py): Captures and maintains a static background model - Segmentation (
segmenter.py,mediapipe_segmenter.py): Foreground/background separation - Guided Filter (
guided_filter.py): Edge-preserving mask refinement - Compositor (
background_compositor.py): Alpha blending and final composition - Camera Application (
ui/camera_app.py): PyQt5-based user interface
- Python 3.12 or higher
- Webcam or camera device
- Windows/Linux/macOS
- Clone the repository:
git clone https://github.com/habibayman/realtime-background-replacement.git
cd realtime-background-replacement- Create a virtual environment (recommended):
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate- Install dependencies:
pip install -r requirements.txtpython main.py- Initialize Background: Click "Initialize Background" and step out of the camera frame for 2 seconds
- Select Background: Choose a background image from the carousel or upload a custom one
- Start Replacement: Click "Start Replacement" to begin real-time processing
- Compare Results: View side-by-side comparison of raw feed, custom pipeline output, and MediaPipe output
Place background images in the assets/ directory, or use the "Upload Custom Background" button in the application to add images dynamically.
realtime-background-replacement/
├── main.py # Application entry point
├── requirements.txt # Python dependencies
├── assets/ # Background images and resources
├── data/ # Data files and samples
├── notebooks/ # Jupyter notebooks for experimentation
└── src/
├── app.py # Alternative entry point
├── config.py # Configuration and hyperparameters
├── pipeline/ # Image processing pipeline modules
│ ├── background_modeling.py # Background capture and modeling
│ ├── segmenter.py # Custom segmentation implementation
│ ├── mediapipe_segmenter.py # MediaPipe-based segmentation
│ ├── guided_filter.py # Edge-aware filtering
│ ├── background_compositor.py # Alpha blending and composition
│ ├── pipeline.py # Custom pipeline orchestration
│ └── ideal_pipeline.py # MediaPipe pipeline orchestration
├── ui/
│ └── camera_app.py # PyQt5 GUI application
└── utils/
└── commonfunctions.py # Utility functions
Core dependencies include:
- OpenCV (
opencv-contrib-python-headless): Image processing and camera capture - PyQt5: GUI framework
- NumPy: Numerical operations
- MediaPipe: Person segmentation (optional, for ideal pipeline)
- Matplotlib: Visualization and analysis
- Pillow: Image I/O operations
See requirements.txt for complete dependency list.
The custom pipeline uses a median-based background model that captures multiple frames of the empty scene and computes a robust background estimate. Segmentation is performed by comparing the current frame against this model in LAB color space.
The guided filter is an edge-preserving smoothing filter that refines the segmentation mask while maintaining sharp boundaries at edges. It uses the input image as a guidance image to transfer structural details to the filtered output.
Final composition uses alpha blending with the refined mask to smoothly composite the foreground subject onto the replacement background, avoiding hard edges and artifacts.
- Target frame rate: ~30 FPS
- Resolution: 640x480 (configurable)
- Camera backend: DirectShow (Windows), V4L2 (Linux), or platform default
- Processing optimizations: NumPy vectorization, efficient color space conversions
- Ensure no other application is using the camera
- Try different camera backends by modifying
camera_idincamera_app.py - Check camera permissions on your operating system
The application handles Qt plugin path conflicts between OpenCV and PyQt5 automatically. If issues persist, ensure you're using compatible versions of both libraries.
If MediaPipe fails to import, the application will fall back to the custom segmentation pipeline. Ensure MediaPipe is properly installed for your platform.
Contributions are welcome. Please follow these guidelines:
- Fork the repository
- Create a feature branch
- Make your changes with clear commit messages
- Test thoroughly
- Submit a pull request
This project is available for educational and research purposes.
- MediaPipe by Google for person segmentation
- Guided Filter implementation based on "Guided Image Filtering" by He et al.
- PyQt5 for the GUI framework
- OpenCV community for computer vision tools
This project was developed by:
Habiba Ayman |
Tasneem Mohammed |
Loay Ahmed |
Omar Gamal |
For questions or issues, please open an issue on the GitHub repository.