Skip to content

PROJECT_STRUCTURE.md

Latest commit

 

History

History
254 lines (209 loc) · 8.34 KB

PROJECT_STRUCTURE.md

File metadata and controls

254 lines (209 loc) · 8.34 KB

Serverless Image Resizer - Project Structure

Repository: https://github.com/Copubah/serverless-image-resizer

Complete Project Layout

serverless-image-resizer/
├── src/
│   └── app.py                          # Main Lambda function for image resizing
├── events/
│   └── s3-event.json                   # Test S3 event for local testing
├── scripts/
│   ├── deploy.sh                       # Automated deployment script
│   ├── test.sh                         # Comprehensive testing script
│   └── cleanup.sh                      # Resource cleanup script
├── enhancements/
│   ├── template-with-dynamodb.yaml     # Enhanced SAM template with DynamoDB
│   ├── app-with-dynamodb.py           # Enhanced Lambda with metadata tracking
│   └── metadata_api.py                 # API for querying image metadata
├── template.yaml                       # Main SAM template
├── requirements.txt                    # Python dependencies
├── samconfig.toml                      # SAM configuration
├── Makefile                           # Build and deployment automation
├── .gitignore                         # Git ignore patterns
├── README.md                          # Comprehensive documentation
└── PROJECT_STRUCTURE.md               # This file

File Descriptions

Core Files

src/app.py

  • Purpose: Main Lambda function for image resizing
  • Features:
    • Processes S3 events automatically
    • Resizes images to 300x300 pixels maintaining aspect ratio
    • Supports JPG, JPEG, PNG, GIF, BMP, TIFF formats
    • Comprehensive error handling and logging
    • Uploads resized images to destination bucket

template.yaml

  • Purpose: AWS SAM template defining infrastructure
  • Resources:
    • Source and destination S3 buckets
    • Lambda function with proper IAM permissions
    • S3 event notifications
    • CloudWatch alarms and log groups
    • SQS Dead Letter Queue
    • Environment-specific parameters

requirements.txt

  • Purpose: Python dependencies
  • Dependencies:
    • boto3==1.34.0 - AWS SDK
    • Pillow==10.1.0 - Image processing library

Configuration Files

samconfig.toml

  • Purpose: SAM CLI configuration
  • Features:
    • Environment-specific deployment settings
    • Build and deployment parameters
    • Multi-environment support (dev/staging/prod)

Makefile

  • Purpose: Build and deployment automation
  • Commands:
    • make install - Install dependencies
    • make validate - Validate SAM template
    • make build - Build application
    • make deploy - Deploy to AWS
    • make test-upload - Upload test image
    • make logs - View Lambda logs
    • make clean - Clean build artifacts
    • make delete - Delete stack

Scripts

scripts/deploy.sh

  • Purpose: Automated deployment with validation
  • Features:
    • Prerequisites checking (AWS CLI, SAM CLI, credentials)
    • Deployment bucket creation
    • Template validation
    • Containerized build
    • Stack deployment with outputs

scripts/test.sh

  • Purpose: Comprehensive testing automation
  • Features:
    • Test image creation
    • Image upload to source bucket
    • Processing verification
    • Dimension validation
    • Local invocation testing
    • Performance metrics collection

scripts/cleanup.sh

  • Purpose: Complete resource cleanup
  • Features:
    • S3 bucket emptying
    • Stack deletion with confirmation
    • Local artifact cleanup
    • Optional deployment bucket cleanup

Test Files

events/s3-event.json

  • Purpose: Sample S3 event for local testing
  • Usage: Used with sam local invoke for testing

Enhancement Files

enhancements/template-with-dynamodb.yaml

  • Purpose: Enhanced SAM template with additional features
  • Additional Resources:
    • DynamoDB table for metadata tracking
    • API Gateway for metadata queries
    • SNS topic for notifications
    • Additional CloudWatch alarms
    • Enhanced monitoring and observability

enhancements/app-with-dynamodb.py

  • Purpose: Enhanced Lambda function with metadata tracking
  • Additional Features:
    • DynamoDB metadata storage
    • SNS notifications
    • Compression ratio calculation
    • Processing time tracking
    • Enhanced error handling

enhancements/metadata_api.py

  • Purpose: REST API for querying image metadata
  • Endpoints:
    • GET /images - List all processed images
    • GET /images/{image_id} - Get specific image metadata
  • Features:
    • Pagination support
    • Filtering by date and source bucket
    • Computed metrics (processing duration, size reduction)

Deployment Environments

Development (dev)

  • Stack Name: serverless-image-resizer-dev
  • Buckets:
    • Source: image-resizer-source-dev-{account-id}
    • Destination: image-resizer-dest-dev-{account-id}
  • Function: image-resizer-dev

Staging (staging)

  • Stack Name: serverless-image-resizer-staging
  • Buckets:
    • Source: image-resizer-source-staging-{account-id}
    • Destination: image-resizer-dest-staging-{account-id}
  • Function: image-resizer-staging

Production (prod)

  • Stack Name: serverless-image-resizer-prod
  • Buckets:
    • Source: image-resizer-source-prod-{account-id}
    • Destination: image-resizer-dest-prod-{account-id}
  • Function: image-resizer-prod

Resource Naming Convention

All resources follow a consistent naming pattern:

  • Format: {service}-{component}-{environment}-{account-id}
  • Examples:
    • S3 Buckets: image-resizer-source-dev-123456789012
    • Lambda Functions: image-resizer-dev
    • DynamoDB Tables: image-metadata-dev
    • CloudWatch Alarms: image-resizer-errors-dev

Security Features

IAM Permissions

  • Principle of Least Privilege: Lambda functions have minimal required permissions
  • Resource-Specific: Permissions scoped to specific buckets and tables
  • Environment Isolation: Each environment has separate resources and permissions

S3 Security

  • Public Access Blocked: All buckets have public access blocked
  • Versioning Enabled: Both source and destination buckets have versioning
  • Encryption: Server-side encryption enabled (can be enhanced with KMS)

Network Security

  • VPC Deployment: Can be deployed in VPC for additional isolation
  • Security Groups: Configurable for VPC deployments
  • Private Subnets: Lambda can be deployed in private subnets

Monitoring and Observability

CloudWatch Metrics

  • Lambda Metrics: Duration, errors, invocations, throttles
  • S3 Metrics: Object counts, bucket sizes
  • DynamoDB Metrics: Read/write capacity, throttling

CloudWatch Alarms

  • Error Alarm: Triggers on function errors > 5 in 10 minutes
  • Duration Alarm: Triggers on average duration > 25 seconds
  • DynamoDB Throttle Alarm: Triggers on throttling events

Logging

  • Structured Logging: JSON-formatted logs with correlation IDs
  • Log Retention: 14-day retention for cost optimization
  • Log Levels: Configurable (INFO, DEBUG, ERROR)

Cost Optimization

Lambda

  • Memory: 512MB for optimal price/performance
  • Timeout: 30 seconds to prevent runaway costs
  • Reserved Concurrency: Limited to 10 to control costs

S3

  • Lifecycle Policies: Can be added for automatic archiving
  • Intelligent Tiering: Recommended for cost optimization
  • Compression: Images are optimized during processing

DynamoDB

  • On-Demand Billing: Pay-per-request pricing
  • Point-in-Time Recovery: Enabled for data protection
  • Auto Scaling: Automatic capacity management

Scalability Considerations

Horizontal Scaling

  • Lambda Concurrency: Configurable reserved concurrency
  • S3 Event Processing: Automatically scales with upload volume
  • DynamoDB: On-demand scaling for variable workloads

Performance Optimization

  • Container Reuse: Lambda containers are reused for efficiency
  • Connection Pooling: AWS SDK connections are reused
  • Image Processing: Optimized Pillow settings for performance

Future Enhancements

Planned Features

  1. Multiple Resize Dimensions: Support for multiple target sizes
  2. WebP Conversion: Modern format support for better compression
  3. Batch Processing: Process multiple images in single invocation
  4. CDN Integration: CloudFront distribution for resized images
  5. Machine Learning: Automatic image tagging and categorization

Integration Options

  1. API Gateway: Direct upload endpoints with presigned URLs
  2. EventBridge: Integration with other AWS services
  3. Step Functions: Complex image processing workflows
  4. SES: Email notifications with processing reports