HLS Video Streaming Platform

A scalable, microservices-based video streaming platform that converts MP4 files to HLS format with adaptive bitrate streaming. Built with Go, Docker, RabbitMQ, MinIO, PostgreSQL, and Redis.

Architecture

┌───────────────────────────────┐
│          Docker Host          │
└────────────────┬──────────────┘
                 │
                 ▼
       ┌───────────────────────┐
       │ docker-compose network│
       └───────────────────────┘
                 │
─────────────────────────────────────────────
│
├── 🧱 upload-service (Go)
│       Accepts MP4, stores to MinIO/input/
│       Publishes to queues:
│          • queue.transcode.480p
│          • queue.transcode.720p
│          • queue.transcode.1080p
│
├── 📨 rabbitmq
│       Hosts the three queues above
│
├── ⚙️ transcoder-worker:480p (Go)
│       Consumes queue.transcode.480p
│       Runs FFmpeg with scale=-2:480
│
├── ⚙️ transcoder-worker:720p (Go)
│       Consumes queue.transcode.720p
│       Runs FFmpeg with scale=-2:720
│
├── ⚙️ transcoder-worker:1080p (Go)
│       Consumes queue.transcode.1080p
│       Runs FFmpeg with scale=-2:1080
│
├── 🗃️ minio
│       S3-compatible storage for input/output
│
├── 🌐 nginx
│       Serves HLS output with caching
│
├── 🧾 postgres
│       Stores job metadata and status
│
└── ⚡ redis
        Fast progress tracking and caching

Features

• Multi-resolution transcoding: Automatically generates 480p, 720p, and 1080p variants
• Adaptive bitrate streaming: HLS with automatic quality switching
• Scalable architecture: Horizontal scaling of transcoder workers
• Queue-based processing: Async job distribution with RabbitMQ
• Progress tracking: Real-time progress updates via Redis
• Caching: NGINX caching layer for efficient content delivery
• Bottleneck mitigation: Resource limits, backpressure handling, optimized I/O
• Comprehensive observability: Prometheus metrics, Grafana dashboards, structured logging

Prerequisites

• Docker Engine 20.10+
• Docker Compose 2.0+
• At least 4 CPU cores and 8GB RAM recommended
• SSD storage for optimal performance

Quick Start

1. Clone and navigate to the project:

   cd HLS

2. Configure environment variables:

   cp .env.example .env
   # Edit .env with your preferred credentials

3. Start all services:

   # Quick start with observability
   ./start-observability.sh
   
   # Or manually
   docker-compose up -d

4. Check service health:

   docker-compose ps

5. Access the platform:

   • Web UI: http://localhost
   • Upload API: http://localhost/api/upload
   • RabbitMQ Management: http://localhost:15672 (user: hls_user, pass: from .env)
   • MinIO Console: http://localhost:9001 (user: minioadmin, pass: from .env)
   • Grafana Dashboard: http://localhost:3000 (admin/admin)
   • Prometheus: http://localhost:9090

Usage

Upload a Video

Via Web UI:

1. Open http://localhost
2. Drag and drop an MP4 file or click to browse
3. Monitor transcoding progress in real-time

Via API:

curl -X POST http://localhost/api/upload \
  -F "file=@your-video.mp4"

Watch a Video

Once transcoding is complete:

http://localhost/player.html?video={video_id}

API Endpoints

• POST /api/upload - Upload MP4 file
• GET /api/videos - List all videos
• GET /api/videos/{id} - Get video details
• GET /api/videos/{id}/status - Get real-time progress

Configuration

Resource Limits

Edit docker-compose.yml to adjust worker resources:

transcoder-worker-1080p:
  deploy:
    resources:
      limits:
        cpus: '2.0'      # Adjust based on your CPU
        memory: 3G       # Adjust based on your RAM

Scaling Workers

Scale specific resolution workers:

# Run 2 instances of 720p worker
docker-compose up -d --scale transcoder-worker-720p=2

# Run 3 instances of 480p worker
docker-compose up -d --scale transcoder-worker-480p=3

FFmpeg Settings

Modify worker environment variables in docker-compose.yml:

environment:
  FFMPEG_THREADS: 2        # Threads per worker
  FFMPEG_PRESET: fast      # fast, medium, slow
  FFMPEG_CRF: 23          # Quality (18-28, lower = better)
  HLS_TIME: 6             # Segment duration in seconds

Monitoring & Observability

Grafana Dashboards

Access http://localhost:3000 (admin/admin) for comprehensive monitoring:

• HLS Platform Overview: System health, request rates, error rates
• Upload Service: Upload metrics, queue depths, API latency
• Transcoder Workers: Job processing, FFmpeg performance, resource usage
• Infrastructure: CPU, memory, disk, network per service
• Logs: Aggregated logs with filters and search

Prometheus Metrics

Access http://localhost:9090 for raw metrics and alerting:

• Custom metrics: hls_upload_requests_total, hls_transcode_jobs_total
• System metrics: CPU, memory, disk usage
• Service health: Database, Redis, RabbitMQ connections
• Alert rules: High queue depth, transcoding failures, resource usage

View Logs

# All services
docker-compose logs -f

# Specific service
docker-compose logs -f upload-service
docker-compose logs -f transcoder-worker-720p

# Structured JSON logs (via Grafana)
# Go to http://localhost:3000 → Explore → Loki

RabbitMQ Management UI

Access http://localhost:15672 to monitor:

• Queue depths
• Message rates
• Consumer status

MinIO Console

Access http://localhost:9001 to view:

• Storage usage
• Bucket contents
• I/O metrics

Troubleshooting

Workers not processing jobs

# Check RabbitMQ connection
docker-compose logs rabbitmq

# Check worker logs
docker-compose logs transcoder-worker-480p

# Restart workers
docker-compose restart transcoder-worker-480p transcoder-worker-720p transcoder-worker-1080p

High CPU usage

# Reduce concurrent workers
docker-compose up -d --scale transcoder-worker-1080p=0

# Or adjust CPU limits in docker-compose.yml

Out of disk space

# Check MinIO usage
docker exec -it hls-minio df -h

# Clean up old videos (implement retention policy)

Development

Project Structure

HLS/
├── services/
│   ├── upload-service/     # Go service for uploads
│   └── transcoder-worker/  # Go service for transcoding
├── web/                    # Frontend HTML/JS
├── config/                 # NGINX, RabbitMQ configs
├── scripts/                # Database init scripts
└── docker-compose.yml      # Service orchestration

Building Services

# Build all services
docker-compose build

# Build specific service
docker-compose build upload-service

Running Tests

# Run Go tests in upload service
cd services/upload-service
go test ./...

# Run Go tests in worker
cd services/transcoder-worker
go test ./...

Performance Optimization

Local Docker Bottlenecks

1. CPU Saturation: Limit concurrent workers, adjust FFmpeg threads
2. I/O Throughput: Use SSD, enable MinIO cache, buffer locally
3. RabbitMQ Backpressure: Set queue limits, implement flow control
4. PostgreSQL Overhead: Use Redis for progress, batch DB writes

See docker-compose.yml for implemented optimizations.

Production Deployment

For production use:

1. Change default passwords in .env
2. Enable HTTPS with SSL certificates
3. Use external PostgreSQL/Redis (managed services)
4. Replace MinIO with AWS S3 or Google Cloud Storage
5. Add monitoring (Prometheus + Grafana)
6. Implement video retention policies
7. Add authentication and authorization
8. Deploy to Kubernetes for better orchestration

License

MIT License - See LICENSE file for details

Contributing

Contributions welcome! Please open an issue or submit a pull request.