Pixy2 Video Servo Controls

A Python-based video servo control system for the Pixy2 camera, enabling automated pan/tilt tracking functionality with audio recording capabilities. This is a hobby project, and a work in progress. Don't use this code if you're not comfortable with c++ and swig, or if you expect it to work correctly on your machine out of the box. thanks!

Next Feature: face recognition + tracking.

Prerequisites

Before building and running the video servo system, ensure you have the following dependencies installed:

macOS

brew install swig libusb portaudio ffmpeg timg

Ubuntu/Debian

sudo apt-get update
sudo apt-get install -y swig libusb-1.0-0-dev python3-dev portaudio19-dev ffmpeg timg

Alternatively, you can use the provided installation script:

./install_prerequisites.sh

Python Dependencies

Install the required Python packages using either:

# From the root directory
pip3 install -r requirements.txt

# Or from the python_demos directory
cd src/python_demos
pip3 install -r requirements.txt

Building

1. Clone the repository:

git clone https://github.com/tamirelazar/video_servo.git
cd video_servo

2. Build the Python module:

cd src/python_demos
python3 setup.py build_ext --inplace

Note: The build process will create a _pixy.cpython-*.so file in the python_demos directory. This file is specific to your Python version and operating system.

Common Build Issues

1. If you get an error about missing libusb-1.0:

   • On macOS: Make sure you've installed libusb with brew install libusb
   • On Ubuntu/Debian: Install with sudo apt-get install libusb-1.0-0-dev

2. If you get an error about missing Python.h:

   • On Ubuntu/Debian: Install python3-dev with sudo apt-get install python3-dev

3. If you get an error about SWIG:

   • Make sure SWIG is installed:
     • macOS: brew install swig
     • Ubuntu/Debian: sudo apt-get install swig

Usage

1. Connect your Pixy2 camera to your computer via USB.

2. Run the video servo program:

cd src/python_demos
python3 video_servo.py

Controls

• Movement:

  • a/d: Pan left/right
  • w/s: Tilt up/down
  • c: Center servos

• Presets:

  • 1: Smooth (slower, precise)
  • 2: Responsive (quick)
  • 3: Precise (balanced speed)
  • 4: Balanced (default)

• Speed Control:

  • [: Decrease velocity scale
  • ]: Increase velocity scale

• Video/Audio:

  • r: Toggle recording (video + audio)
  • p: Toggle display mode (terminal/window)
  • q: Quit

Display Modes

The system supports two display modes for the video preview:

• Terminal Display (default): Displays the video directly in the terminal using timg. This mode is useful for remote connections or when you want to keep everything in a single terminal window.
• Window Display: Shows the video in a separate OpenCV window.

Toggle between these modes using the p key.

Audio Recording

The system now supports audio recording alongside video capture. When enabled in the config.json file, audio will be recorded from your selected microphone whenever video recording is active. The recordings are automatically combined using FFmpeg after recording stops.

To configure audio settings, modify the audio section in src/config.json:

"audio": {
    "enabled": true,       // Set to false to disable audio recording
    "sample_rate": 44100,  // Audio sample rate in Hz
    "channels": 1,         // Number of audio channels (1 for mono, 2 for stereo)
    "format": "paInt16",   // Audio format (paInt16, paInt24, paInt32, paFloat32)
    "chunk_size": 1024,    // Audio buffer chunk size
    "device_index": null   // Specify a device index or null for default
}

On first run with audio enabled, you'll be prompted to select an audio input device from available microphones.

Output Files

Recordings are saved to the src/output directory with the following naming convention:

• Video: video_servo_YYYYMMDD_HHMMSS.mp4
• Audio: audio_servo_YYYYMMDD_HHMMSS.wav
• Combined: video_servo_YYYYMMDD_HHMMSS_combined.mp4

Project Structure

.
├── src/
│   ├── arduino/            # Arduino library headers
│   ├── common/            # Common utility files
│   ├── libpixyusb2/      # Core USB communication library
│   ├── audio_recorder.py  # Audio recording functionality
│   ├── config.json        # Configuration settings
│   ├── video_servo_interface.py # Main interface program
│   └── python_demos/     # Python interface and examples
│       ├── setup.py      # Build script for Python module
│       ├── pixy.i        # SWIG interface file
│       ├── pixy_python_interface.cpp  # Python/C++ interface
│       ├── requirements.txt # Python dependencies
│       └── video_servo.py # Main program
├── install_prerequisites.sh
└── README.md

Development

The project uses SWIG to generate Python bindings for the C++ Pixy2 library. The main components are:

• video_servo.py: Main program for video-based servo control
• libpixyusb2: Core C++ library for USB communication with Pixy2
• SWIG interface files for Python bindings

Building from Source

The build process involves several steps:

1. SWIG generates Python wrapper code from the interface file (pixy.i)
2. The C++ code is compiled into a shared library
3. Python bindings are created to interface with the library

If you modify any of the C++ source files or SWIG interface, you'll need to rebuild the module:

cd src/python_demos
python3 setup.py clean --all
python3 setup.py build_ext --inplace

License

This project is derived from the Pixy2 library. All original Pixy2 code retains its original license.

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

Audio-Video Synchronization

• Timestamp-Based Synchronization: Each audio and video frame is precisely timestamped at capture time
• Real-time Synchronized Access: External applications can access perfectly synchronized audio and video streams
• Drift Monitoring: The system continuously monitors audio-video drift with visual indicators
• Enhanced FFmpeg Processing: Improved audio-video combination that uses timestamp metadata
• Configurable Parameters: Fine-tune synchronization behavior through config.json
• Adaptive Time Stretching: Automatically stretches the shorter media file (audio or video) to match the duration of the longer one, ensuring perfect synchronization

Configuration

Configure synchronization in the config.json