Project SEAL-See All is a comprehensive computer vision application designed to augment the interaction between users and computers using facial detection, hand gesture recognition, and object detection. Implemented on a macOS environment and leveraging a webcam, SEAL enables users to control actions or trigger scripts through specific hand gestures while also offering object detection and text extraction capabilities.
Features Facial Detection and Recognition: Identifies and recognizes faces in real-time video streams. Hand Gesture Recognition: Detects hand gestures including closed fist, open palm, pointing up, thumbs up, thumbs down, and victory sign using MediaPipe. Object Detection: Utilizes YOLO COCO dataset for recognizing a wide range of common objects. Optical Character Recognition (OCR): Extracts text from identified objects with high confidence using PyTesseract.
Installation Ensure you have Python 3.x installed on your macOS. Clone the repository and navigate to the project directory.
git clone cd SEAL-See-All
Dependencies
Install the required Python packages: pip install -r requirements.txt
Make sure to have the models and required files in the following paths: YOLO configuration and weights: /yolo/yolov3.cfg, /yolo/yolov3.weights COCO names: /yolo/coco.names Haar Cascade for face detection: /haar/haarcascade_frontalface_default.xml Gesture recognizer model: gesture_recognizer/gesture_recognizer.task Tesseract OCR: Ensure tesseract is installed and correctly set up on your macOS.
Preparing Facial Encodings Place sample images in the designated folder. Run image_augmentation.py to augment your images for better recognition performance. Use face_encoder.py to generate facial encoding files.
python image_augmentation.py python face_encoder.py
Usage Run the main script to start the application:
python seal.py
Toggle Object Detection: Press the 'o' key to turn on/off object detection. Extract Text: Press the 'a' key to activate OCR on objects detected with over 80% confidence.
Yolo Objects:
names:
0: person
1: bicycle
2: car
3: motorcycle
4: airplane
5: bus
6: train
7: truck
8: boat
9: traffic light
10: fire hydrant
11: stop sign
12: parking meter
13: bench
14: bird
15: cat
16: dog
17: horse
18: sheep
19: cow
20: elephant
21: bear
22: zebra
23: giraffe
24: backpack
25: umbrella
26: handbag
27: tie
28: suitcase
29: frisbee
30: skis
31: snowboard
32: sports ball
33: kite
34: baseball bat
35: baseball glove
36: skateboard
37: surfboard
38: tennis racket
39: bottle
40: wine glass
41: cup
42: fork
43: knife
44: spoon
45: bowl
46: banana
47: apple
48: sandwich
49: orange
50: broccoli
51: carrot
52: hot dog
53: pizza
54: donut
55: cake
56: chair
57: couch
58: potted plant
59: bed
60: dining table
61: toilet
62: tv
63: laptop
64: mouse
65: remote
66: keyboard
67: cell phone
68: microwave
69: oven
70: toaster
71: sink
72: refrigerator
73: book
74: clock
75: vase
76: scissors
77: teddy bear
78: hair drier
79: toothbrush
Adding Gesture Actions
Edit the gesture_recognition_callback function in seal.py to define actions for detected gestures.
To-Do:
Image Enhancement Integration:
I plan to integrate an image enhancement model to refine the images captured for object detection. This enhancement aims to significantly improve the quality of images before they are processed by OCR, thereby enhancing the accuracy of text extraction from complex backgrounds or low-quality images.
Voice Command Activation:
In an effort to broaden the interactivity of SEAL-See All, I am working on incorporating a feature that enables live voice recording with immediate transcription. The transcribed audio will then be fed into ShellGPT, facilitating the conversion of voice commands into actionable code execution within the command line environment. This enhancement is targeted towards creating a more seamless and hands-free user experience, enabling users to execute commands and control the application through voice interactions.
Stay tuned for these exciting updates as we continue to enhance the functionality and user experience of Project SEAL-See All.