Update Docs

Author: alokemajumder

DOCKER_QUICKSTART.md

@@ -11,17 +11,17 @@
 ### 2️⃣ Quick Setup
 ```bash
 # Clone repository
-git clone <repository-url>
+git clone https://github.com/your-username/Video-subtitle-Generator.git
 cd Video-subtitle-Generator
 
-# Create data folders
-mkdir -p data/{input,output,config,logs,temp,jobs}
+# Run automated setup (recommended)
+./setup.sh
 
-# Copy your Google Cloud credentials
-cp /path/to/your-service-account.json data/config/
-
-# Make scripts executable (Linux/Mac)
-chmod +x docker-run.sh docker-entrypoint.sh
+# OR manual setup:
+mkdir -p input output logs temp jobs
+cp /path/to/your-service-account.json ./service-account.json
+cp .env.template .env
+# Edit .env with your Google Cloud settings
 ```
 
 ### 3️⃣ Run It!
@@ -43,7 +43,7 @@ That's it! The application will build and start in interactive mode.
 ### Method 1: Interactive Mode (Easiest)
 ```bash
 # Copy video to input folder
-cp my-video.mp4 data/input/
+cp my-video.mp4 input/
 
 # Run interactive mode
 ./docker-run.sh              # Linux/Mac
@@ -54,12 +54,14 @@ docker-run.bat               # Windows
 
 ### Method 2: Command Line
 ```bash
-# Modern docker compose syntax
-docker compose run --rm subtitle-generator python main.py --video /data/input/my-video.mp4 --languages eng,hin,ben
+# Single video with precision validation
+docker compose run --rm subtitle-generator python main.py --video input/my-video.mp4 --languages eng,hin,ben
+
+# Batch process all videos in input/
+docker compose run --rm subtitle-generator python main.py --batch input/
 
-# Or using convenience scripts
-./docker-run.sh --video /data/input/my-video.mp4 --languages eng,hin,ben,tel
-./docker-run.sh --batch /data/input
+# Generate SDH (accessibility) subtitles
+docker compose run --rm subtitle-generator python main.py --video input/my-video.mp4 --languages eng --sdh
 ```
 
 ## 🎮 Common Commands

PRODUCTION.md

@@ -35,8 +35,8 @@ The system has been enhanced with production-grade features:
 ### 1. System Requirements
 
 ```bash
-# Check system requirements
-python3 test_production.py
+# Test the system setup
+python3 test_precision_subtitles.py
 ```
 
 **Minimum Requirements:**

README.md

@@ -32,37 +32,50 @@ An enterprise-grade AI-powered subtitle generation system using Google Gemini AI
 
 ### 1️⃣ Setup
 ```bash
-git clone <repository-url>
+git clone https://github.com/your-username/Video-subtitle-Generator.git
 cd Video-subtitle-Generator
 
-# Create data directories
-mkdir -p data/{input,output,config,logs,temp,jobs}
+# Run automated setup (recommended)
+./setup.sh
 
-# Add your Google Cloud credentials
-cp /path/to/your-service-account.json data/config/
+# OR manually create directories and configure
+mkdir -p input output logs temp jobs
+cp /path/to/your-service-account.json ./service-account.json
+cp .env.template .env
+# Edit .env with your Google Cloud settings
 ```
 
-### 2️⃣ Run
+### 2️⃣ Verify Setup
 ```bash
-# Modern Docker Compose syntax (uses compose.yml)
+# Test Docker configuration
+docker compose config
+
+# Verify all components
+docker compose run --rm subtitle-generator python -c \
+  "from src.config_manager import ConfigManager; print('✅ Setup OK!' if ConfigManager().validate_setup() else '❌ Setup issues')"
+```
+
+### 3️⃣ Run
+```bash
+# Interactive mode (recommended for first time)
 docker compose run --rm subtitle-generator
 
 # Or use convenience scripts
 ./docker-run.sh              # Linux/Mac
 docker-run.bat               # Windows
 ```
 
-### 3️⃣ Process Videos
+### 4️⃣ Process Videos
 ```bash
 # Copy videos to input
-cp your-video.mp4 data/input/
+cp your-video.mp4 input/
 
-# Process interactively (select option 1)
+# Process interactively (recommended)
 docker compose run --rm subtitle-generator
 
-# Or process directly
+# Or process directly with CLI
 docker compose run --rm subtitle-generator \
-  python main.py --video /data/input/your-video.mp4 --languages eng,hin
+  python main.py --video input/your-video.mp4 --languages eng,hin,ben
 ```
 
 ## 🎯 Usage Examples
@@ -77,15 +90,15 @@ docker compose run --rm subtitle-generator
 ```bash
 # Single video with core + Indian languages
 docker compose run --rm subtitle-generator \
-  python main.py --video /data/input/movie.mp4 --languages eng,hin,ben,tel,tam
+  python main.py --video input/movie.mp4 --languages eng,hin,ben,tel,tam
 
-# Batch process all videos
+# Batch process all videos in input directory
 docker compose run --rm subtitle-generator \
-  python main.py --batch /data/input
+  python main.py --batch input/
 
 # Generate accessibility subtitles (SDH)
 docker compose run --rm subtitle-generator \
-  python main.py --video /data/input/video.mp4 --languages eng --sdh
+  python main.py --video input/video.mp4 --languages eng --sdh
 
 # Resume interrupted job
 docker compose run --rm subtitle-generator \
@@ -117,70 +130,87 @@ Video-subtitle-Generator/
 │   ├── main.py                    # Entry point
 │   ├── src/                       # Core application
 │   │   ├── subtitle_processor.py  # Main processing logic
-│   │   ├── ai_generator.py        # Gemini AI integration
+│   │   ├── ai_generator.py        # Gemini AI integration + translation
+│   │   ├── precision_validator.py # Quality validation system
+│   │   ├── translation_quality_analyzer.py # Cross-language quality
 │   │   ├── gcs_handler.py         # Cloud Storage
 │   │   └── ...                    # Other components
-│   └── config/                    # Configuration files
-└── 📊 Data (Created at runtime)
-    ├── data/input/                # Place videos here
-    ├── data/output/               # Find subtitles here
-    ├── data/config/               # service-account.json
-    └── data/logs/                 # Application logs
+│   └── config/                    # Configuration files & AI prompts
+├── 📊 Working Directories (Created by setup.sh)
+│   ├── input/                     # Place videos here
+│   ├── output/                    # Find subtitles here (SRT & VTT)
+│   ├── logs/                      # Application logs
+│   ├── temp/                      # Temporary processing files
+│   └── jobs/                      # Job state files
+├── 🔧 Configuration
+│   ├── service-account.json       # Your Google Cloud credentials
+│   ├── .env                       # Environment configuration
+│   └── .env.template             # Configuration template
 ```
 
 ## ⚙️ Configuration
 
-### Custom Settings
-Create `data/config/config.yaml`:
+### Environment Configuration
+Copy and edit the environment template:
+```bash
+cp .env.template .env
+# Edit .env with your settings
+```
+
+Key settings in `.env`:
+```bash
+GCP_PROJECT_ID=your-gcp-project-id
+GCP_LOCATION=us-central1
+GOOGLE_APPLICATION_CREDENTIALS=./service-account.json
+VERTEX_AI_MODEL=gemini-2.5-pro-preview-05-06
+MIN_TRANSLATION_QUALITY=0.70   # Translation quality threshold
+MIN_CULTURAL_ACCURACY=0.80     # Cultural accuracy threshold
+```
+
+### Advanced Configuration
+Edit `config/config.yaml` for fine-tuning:
 ```yaml
 vertex_ai:
   temperature: 0.2              # AI creativity (0.0-1.0)
   max_output_tokens: 8192       # Response length limit
+  model: "gemini-2.5-pro-preview-05-06"
 
 processing:
   chunk_duration: 60            # Video chunk size (seconds)
-  parallel_workers: 4           # Concurrent processing
-  max_retries: 3               # Error retry attempts
-```
-
-### Environment Variables
-Edit `compose.yml`:
-```yaml
-environment:
-  LOG_LEVEL: INFO               # DEBUG, INFO, WARNING, ERROR
-  ENV: production               # production, development
+  max_concurrent_jobs: 3        # Parallel processing limit
+  max_retry_attempts: 3         # Quality-driven retries
+
+# NEW: Translation quality settings
+translation_quality:
+  enable_validation: true       # Enable cross-language validation
+  min_bleu_score: 0.25         # Minimum BLEU score
+  min_cultural_accuracy: 0.80   # Minimum cultural score
 ```
 
 ## 🌍 Supported Languages
 
-### 🔑 Core Languages (Mandatory Support)
-| Code | Language | Method |
-|------|----------|---------|
-| `eng` | English | Direct transcription |
-| `hin` | Hindi | Dual (transcription + translation) |
-| `ben` | Bengali | Direct transcription |
+### 🔑 Core Languages (Precision Quality)
+| Code | Language | Features |
+|------|----------|----------|
+| `eng` | English | ✅ Direct transcription, Human-level validation |
+| `ben` | Bengali | ✅ Direct transcription, Cultural context validation |
+| `hin` | Hindi | ✅ Dual method (direct + translation), Devanagari accuracy |
 
-### 🇮🇳 Optional Indian Languages
+> **Note**: Core languages feature **precision validation** with 95%+ accuracy, **translation quality assessment**, and **cultural context preservation**.
+
+### 🇮🇳 Supported Indian Languages  
 | Code | Language | Method |
 |------|----------|---------|
-| `tel` | Telugu | Translation from core languages |
-| `mar` | Marathi | Translation from core languages |
-| `tam` | Tamil | Translation from core languages |
-| `guj` | Gujarati | Translation from core languages |
-| `kan` | Kannada | Translation from core languages |
-| `mal` | Malayalam | Translation from core languages |
-| `pun` | Punjabi | Translation from core languages |
-| `ori` | Odia | Translation from core languages |
-| `asm` | Assamese | Translation from core languages |
-| `urd` | Urdu | Translation from core languages |
-| `san` | Sanskrit | Translation from core languages |
-| `kok` | Konkani | Translation from core languages |
-| `nep` | Nepali | Translation from core languages |
-| `sit` | Sinhala | Translation from core languages |
-| `mai` | Maithili | Translation from core languages |
-| `bho` | Bhojpuri | Translation from core languages |
-| `raj` | Rajasthani | Translation from core languages |
-| `mag` | Magahi | Translation from core languages |
+| `tel` | Telugu | AI transcription/translation |
+| `tam` | Tamil | AI transcription/translation |
+| `mar` | Marathi | AI transcription/translation |
+| `guj` | Gujarati | AI transcription/translation |
+| `kan` | Kannada | AI transcription/translation |
+| `mal` | Malayalam | AI transcription/translation |
+| `pun` | Punjabi | AI transcription/translation |
+| `urd` | Urdu | AI transcription/translation |
+
+**Usage**: `--languages eng,hin,ben,tel,tam` (mix and match as needed)
 
 ## 📊 Health Monitoring
 
@@ -203,7 +233,7 @@ docker stats subtitle-generator
 docker compose logs -f subtitle-generator
 
 # Error tracking
-docker compose exec subtitle-generator cat /data/logs/errors.jsonl
+docker compose exec subtitle-generator cat logs/errors.jsonl
 ```
 
 ## 🚨 Troubleshooting
@@ -212,10 +242,12 @@ docker compose exec subtitle-generator cat /data/logs/errors.jsonl
 
 | Problem | Solution |
 |---------|----------|
-| "No service account found" | Copy `service-account.json` to `data/config/` |
-| "Permission denied" | `sudo chown -R $USER:$USER data/` (Linux/Mac) |
-| "Out of memory" | Increase Docker memory to 8GB+ |
+| "No service account found" | Place `service-account.json` in project root |
+| "Permission denied" | `sudo chown -R $USER:$USER .` (Linux/Mac) |
+| "Out of memory" | Increase Docker memory to 8GB+ in Docker Desktop |
 | "Cannot connect to Docker" | Ensure Docker Desktop is running |
+| "Translation quality too low" | Video audio may be unclear or multilingual |
+| "Module not found" | Run `./setup.sh` to ensure proper setup |
 
 ### Debug Mode
 ```bash
@@ -268,11 +300,22 @@ gcloud run deploy subtitle-generator \
 
 ## 📈 Performance Metrics
 
-- **⚡ Processing Speed**: ~1x real-time for single language
-- **🎯 Accuracy**: 95%+ for clear audio content
-- **💾 Memory Usage**: 2-8GB depending on video size and settings
-- **🔄 Throughput**: Configurable parallel processing (1-8 workers)
-- **📊 Reliability**: 99.9% uptime with proper error handling
+### 🚀 Processing Performance
+- **⚡ Speed**: ~1-2x real-time per language (depends on video quality)
+- **🎯 Accuracy**: 95%+ for core languages with precision validation
+- **💾 Memory**: 4-8GB recommended (2GB minimum)
+- **🔄 Throughput**: Up to 3 concurrent jobs (configurable)
+
+### 💯 Quality Metrics (NEW)
+- **Translation Quality**: 70%+ BLEU score for production
+- **Cultural Accuracy**: 80%+ for Bengali/Hindi cultural context  
+- **Fluency Score**: 80%+ target language naturalness
+- **Retry Success**: 90%+ quality improvement on retry
+
+### 📊 Reliability
+- **Error Recovery**: Automatic retry with quality validation
+- **Format Support**: SRT + VTT dual output
+- **Resource Management**: Automatic cleanup and monitoring
 
 ## 🔗 Documentation