Image Processing AWS Pipeline

A production-grade, highly secure, and fully decoupled image processing infrastructure on AWS built with Terraform.

This repository implements a reference architecture featuring a VPC-isolated application tier (ALB + private EC2 Auto Scaling Group), a private RDS database, S3 event-driven Lambda image processing, and centralized security & monitoring (KMS, IAM, Secrets Manager, and CloudWatch).

---

🏗️ Architecture

Workflow Diagram

graph TD
    Client[1. Client] -->|HTTP Request| ALB[2. Application Load Balancer]
    ALB -->|Route Traffic| EC2[3. Auto Scaling Group]
    
    subgraph PrivateSubnets ["Private Subnets (Isolated)"]
        EC2 -->|Read/Write Metadata| RDS[(4. RDS PostgreSQL)]
    end

    EC2 -->|5. Upload Raw Image| S3[6. S3 Bucket /uploads]
    S3 -->|7. ObjectCreated Event| Lambda[8. Lambda Processor]
    Lambda -->|9. Save Processed Image| S3Processed[S3 Bucket /processed]

    subgraph SecurityMonitoring ["Security & Monitoring"]
        KMS[KMS Key] -.->|Encrypts| RDS
        KMS -.->|Encrypts| S3
        KMS -.->|Encrypts| Secrets[Secrets Manager]
        CloudWatch[CloudWatch Alarms] -.->|Alerts| SNS[SNS Alerts Topic]
    end

Loading

---

📂 Repository Layout

image-processing-aws-pipeline/
├── .github/
│   └── workflows/
│       └── terraform.yml      # CI pipeline (Lint check, validate & security scan)
├── images/
│   └── architecture.png       # AWS Architecture diagram
├── modules/                   # Standalone, reusable modules (zero cross-module dependencies)
│   ├── vpc/                   # VPC, public/private subnets, IGW, NAT Gateway
│   ├── alb/                   # Application Load Balancer, target group, listener
│   ├── ec2/                   # Launch Template + Auto Scaling Group (enforces IMDSv2 & KMS)
│   ├── rds/                   # RDS PostgreSQL instance & secure DB subnet/security groups
│   ├── s3/                    # S3 bucket configuration (versioning, encryption, lifecycle rules)
│   ├── lambda/                # Lambda function packaging & execution permissions
│   ├── iam/                   # IAM role & policy factory
│   ├── kms/                   # Customer-managed KMS key, policy, and alias
│   ├── secrets-manager/       # Secrets Manager secret wrapper
│   └── cloudwatch/            # CloudWatch metric alarms (ASG, ALB, RDS, Lambda)
├── examples/
│   └── complete/              # Wires modules together into a complete pipeline
│       ├── src/lambda/        # Sample image-processing Lambda source (Python)
│       ├── main.tf            # Main entrypoint configuring all modules
│       ├── variables.tf       # Parameter declarations
│       └── outputs.tf         # Outputs (ALB URL, Bucket name, Secret ARNs, etc.)
├── scripts/
│   └── simulate_pipeline.py   # E2E pipeline tester (S3 uploads, log-streaming, load test)
├── .env.example               # Template for AWS credentials & Terraform variables
└── .gitignore                 # Configured to ignore .env and local Terraform states

---

🛡️ Production Design & Security Principles

• Fully Decoupled Modules: Every module in modules/ is independent. They take standard inputs and return standard outputs. You can reference them directly from this Git repository in other projects.
• VPC Isolation: The Application Load Balancer (ALB) sits in public subnets, whereas the EC2 Application hosts and RDS Database reside in private subnets, fully isolated from the public internet.
• Zero Hardcoded Secrets: Master database credentials are dynamically generated using random_password and stored inside AWS Secrets Manager, encrypted using a custom KMS key.
• Encryption Everywhere: Data is encrypted at rest using a shared Customer Managed Key (KMS CMK) across S3, RDS, EBS root volumes, and Secrets Manager.
• CI/CD Verified: Continuous Integration using GitHub Actions validates Terraform formatting (terraform fmt), syntax rules (terraform validate), and scans for security misconfigurations.

---

🚀 Getting Started & Deployment

Step 1: Clone the Repository

git clone https://github.com/samrato/image-processing-aws-pipeline.git
cd image-processing-aws-pipeline

Step 2: Configure Environment Variables

Copy .env.example to .env and fill in your AWS credentials:

cp .env.example .env
# Edit '.env' with your AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, etc.

Load the variables into your terminal session:

source .env

Step 3: Deploy the Infrastructure

Change directory to the complete example and deploy:

cd examples/complete

# Initialize Terraform (downloads providers & modules)
terraform init

# Plan and review the infrastructure changes
terraform plan

# Apply changes to provision resources in AWS
terraform apply

---

📱 Testing & Pipeline Simulation

Once your pipeline is successfully applied, you can run the interactive testing script at the root of the repository to simulate load and events.

Make sure your AWS environment variables are sourced, then run:

# Install dependencies
pip install boto3 requests

# Run the interactive simulator
./scripts/simulate_pipeline.py

Simulation Modes:

1. Run Diagnostics: Validates that all provisioned S3 buckets, ALB endpoints, and Lambda functions are active.
2. Upload Test Images: Uploads sample files to the S3 bucket to trigger the Lambda processing event.
3. View Lambda Logs: Streams active CloudWatch logs to verify the Lambda function is running.
4. Run ALB HTTP Load Test: Sends high-volume traffic through the ALB to trigger the Auto Scaling policy.

---

🧹 Cleaning Up

To tear down all resources and avoid running up costs on AWS, run:

cd examples/complete
terraform destroy