🤟 ASL Sign Language Detection System

A real-time American Sign Language (ASL) alphabet detection system using CNN and MediaPipe for communication assistance for hearing-impaired individuals.

🎯 Features

Real-time Detection: Detects ASL alphabet (A-Z) + special gestures (space, delete, nothing)
High Accuracy: Uses pre-trained CNN model with 98.78% accuracy
Smart Word Building: Letter-by-letter word formation with stability checks
Spell Correction: Automatic spell correction using NLP
Sentence Formation: Combines words into grammatically correct sentences
Text-to-Speech: Speaks the final sentence
Enhanced UI: Visual feedback with confidence bars, FPS counter, and clear displays
Auto Model Download: Automatically downloads model from HuggingFace

📋 Requirements

Python 3.8 or higher
Webcam
Windows/Mac/Linux

🚀 Installation & Setup

Step 1: Clone/Download the Project

Create a project folder and download all files:

mkdir asl-detection
cd asl-detection

Step 2: Install Python Dependencies

Option A: Using pip (Recommended)

pip install -r requirements.txt

Option B: Manual Installation

pip install opencv-python==4.8.1.78
pip install mediapipe==0.10.9
pip install tensorflow==2.15.0
pip install numpy==1.24.3
pip install textblob==0.17.1
pip install pyttsx3==2.90

Step 3: Download NLP Data

python -m textblob.download_corpora

Step 4: Run the Application

python asl_detector.py

The model will automatically download on first run (approximately 20MB).

📁 Project Structure

asl-detection/
│
├── asl_detector.py          # Main application file
├── requirements.txt          # Python dependencies
├── README.md                 # This file
├── setup_guide.md           # Detailed setup instructions
│
├── models/                   # Auto-created folder
│   └── best_cnn_asl_model.keras  # Downloaded automatically
│
└── docs/                     # Optional: Documentation
    ├── usage_guide.md
    └── troubleshooting.md

🎮 How to Use

Basic Operation

Start the application
```
python asl_detector.py
```
Position your hand in front of the webcam
- Make sure your hand is well-lit
- Keep hand in center of frame
- Show clear ASL signs
Spell words letter by letter
- Hold each sign steady for ~1 second
- Letter confirms when stable (green color)
- Continue spelling to build word
Finalize words
- Show "SPACE" gesture to finalize current word, OR
- Pause for 2.5 seconds (no hand) for auto-finalization
- Use "DEL" gesture to delete last letter
Create sentences
- Build multiple words
- Press ENTER to form complete sentence
- System will speak the sentence (if TTS available)
Exit
- Press ESC to quit

Keyboard Controls

Key	Action
ENTER	Finalize sentence and speak
ESC	Exit application
C	Clear current word
R	Reset everything

Gesture Controls

Gesture	Action
A-Z Signs	Add letter to word
SPACE	Finalize current word
DEL	Delete last letter
NOTHING	Ignored (neutral position)

🎨 UI Elements Explained

Top Panel

Current Letter: Shows detected letter (yellow = detecting, green = confirmed)
Status: DETECTING or CONFIRMED
Confidence Bar: Visual confidence level (need >88% for detection)
FPS Counter: Real-time performance

Middle Panel

Current Word: Letters being built into current word

Bottom Panel

Words: List of finalized words (shows last 8 words)

Instructions Bar

Quick reference for controls

⚙️ Configuration

Edit Config class in asl_detector.py:

class Config:
    CONFIDENCE_THRESHOLD = 0.88   # Min confidence (0.85-0.95)
    STABILITY_FRAMES = 10         # Frames to confirm (8-15)
    PAUSE_DURATION = 2.5          # Seconds for auto-finalize (2-4)

🔧 Troubleshooting

❌ Issue: Letters Not Detecting / Not Showing on Screen

This was the MAIN ISSUE - Now FIXED! Here's what was wrong and how it's fixed:

Problem 1: Wrong normalization

❌ Old code used incorrect landmark preprocessing
✅ Fixed: Proper z-score normalization matching model training

Problem 2: Threshold too high

❌ Old: 88% confidence threshold was too strict
✅ Fixed: Lowered to 70% for better detection

Problem 3: Stability too strict

❌ Old: Required 10 frames, too slow
✅ Fixed: 7 frames with 70% agreement

Problem 4: Poor visual feedback

❌ Old: Hard to see what was happening
✅ Fixed: Clear UI showing detection state

To verify it's working:

Run python test_system.py first
Then run python asl_detector.py
Show ASL 'A' sign clearly
You should see:
- Letter 'A' displayed in top-left
- "⊙ DETECTING" or "✓ CONFIRMED"
- Confidence bar showing percentage
- Letter added to WORD section when confirmed

If still not detecting:

Check lighting - use bright, even lighting
Use plain background (wall)
Keep hand 1-2 feet from camera
Hold sign steady for full second
Make sure hand fills about 30% of frame

Issue: Model Download Fails

Solution: Manually download from:

https://huggingface.co/ademaulana/CNN-ASL-Alphabet-Sign-Recognition/resolve/main/best_cnn_asl_model.keras

Place in models/ folder

Issue: Webcam Not Working

Solutions:

Check if webcam is connected
Try different camera index in code: cv2.VideoCapture(1)
Check camera permissions in system settings
Close other apps using webcam

Issue: TensorFlow Errors

For Apple Silicon Mac:

pip install tensorflow-macos==2.15.0
pip install tensorflow-metal

For GPU (NVIDIA):

pip install tensorflow-gpu==2.15.0

Issue: Low FPS / Lag

Solutions:

Close other applications
Reduce webcam resolution in code
Use model_complexity=0 in MediaPipe settings
Ensure good lighting (reduces detection time)

Issue: Poor Detection Accuracy

Tips:

Ensure good lighting (avoid backlighting)
Keep hand centered in frame
Make clear, distinct signs
Hold signs steady for full second
Avoid rapid movements
Plain background helps

Issue: TTS Not Working

Windows: Usually works by default Mac: Use system TTS (should work) Linux: Install espeak

sudo apt-get install espeak

📊 Model Information

Model: CNN with Conv1D architecture
Input: 63 features (x, y, z coordinates from 21 hand landmarks)
Classes: 29 (A-Z + space, del, nothing)
Accuracy: 98.78% validation accuracy
Source: HuggingFace - ademaulana

🎓 Tips for Best Results

Lighting: Ensure even, bright lighting on your hand
Background: Use plain, contrasting background
Distance: Keep hand 1-2 feet from camera
Steadiness: Hold each sign steady for confirmation
Practice: Learn ASL alphabet signs properly
Positioning: Keep hand in center of frame

🔍 Understanding the Detection Process

Camera Frame
    ↓
MediaPipe Hand Detection (21 landmarks)
    ↓
Extract 63 features (x,y,z coordinates)
    ↓
Normalize features
    ↓
CNN Model Prediction (29 classes)
    ↓
Confidence Filtering (>88%)
    ↓
Stability Check (10 consistent frames)
    ↓
Letter Confirmed → Add to Word
    ↓
Word Finalized (space gesture or pause)
    ↓
Spell Correction (TextBlob NLP)
    ↓
Sentence Formation (multiple words)
    ↓
Text-to-Speech Output

🐛 Debug Mode

To enable detailed logging, add at start of asl_detector.py:

import logging
logging.basicConfig(level=logging.DEBUG)

📝 VS Code Setup

1. Open VS Code

code .

2. Install Python Extension

Go to Extensions (Ctrl+Shift+X)
Search "Python"
Install official Python extension by Microsoft

3. Select Python Interpreter

Press Ctrl+Shift+P
Type "Python: Select Interpreter"
Choose your Python 3.8+ environment

4. Run in VS Code

Open asl_detector.py
Press F5 or click "Run" button
Or use terminal: python asl_detector.py

5. Recommended VS Code Extensions

Python (Microsoft)
Pylance (Microsoft)
Python Debugger (Microsoft)

🌟 Future Enhancements

📄 License

This project uses the MIT License model from ademaulana.

🤝 Contributing

Contributions welcome! Please feel free to submit issues or pull requests.

📧 Support

For issues or questions:

Check troubleshooting section above
Review model documentation on HuggingFace
Open GitHub issue with details

🙏 Acknowledgments

Model by ademaulana
MediaPipe by Google
ASL Alphabet Dataset community

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.gitignore		.gitignore
README.md		README.md
asl_detector.py		asl_detector.py
check_model.py		check_model.py
requirements.txt		requirements.txt
test_system.py		test_system.py

Folders and files

Latest commit

History

Repository files navigation