AWS EC2 Inference Solution - Customization Guide

This document provides detailed instructions for customizing the inference solution to meet your specific requirements. The solution has been designed with modularity in mind, allowing for easy customization of various components.

Customizing the Application

Modifying the API

The sample application is located in the app/ directory. To customize the application:

Modify server.js
- Update the routes and logic according to your needs
- Add new endpoints for your specific inference requirements
- Implement authentication/authorization as needed
Add Dependencies
- Update package.json to include any additional libraries
- Install dependencies with npm install
Test Locally
```
cd app
npm install
npm start
```
Update the Dockerfile
- Change the base image if needed
- Add any additional build steps
- Optimize for production use
Rebuild and Deploy
- Changes will be automatically built and deployed when running terraform apply

Implementing a Different Backend

To replace the Node.js application with a different backend:

Create a New Application Directory
- Create a new directory for your application code
- Implement your API with your preferred language/framework
Create a New Dockerfile
- Define the build process for your application
- Ensure it exposes the appropriate port
- Include any necessary environment variables
Update the Build Module
- Modify modules/build/main.tf to point to your new application directory
- Update the file hash triggers to match your application files
Update the EC2 Module
- Adjust modules/ec2/templates/inference-app.service.tpl if needed
- Update ports and environment variables

Customizing Infrastructure

Instance Type and Size

To change the EC2 instance type:

Update terraform.tfvars:

instance_type = "t3.medium"  # Choose the appropriate instance type

Adjust the root volume size in terraform.tfvars or variables.tf:

# In variables.tf
variable "root_volume_size" {
  description = "Size of the root volume in GB"
  type        = number
  default     = 50  # Increase from the default 30GB
}

Networking Configuration

To customize the VPC and networking:

Change CIDR Ranges:

# In terraform.tfvars
vpc_cidr = "10.1.0.0/16"  # Set a custom CIDR range

Modify Subnet Configuration: Edit main.tf to change the subnet allocation:

private_subnets = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k)]
public_subnets  = [for k, v in local.azs : cidrsubnet(var.vpc_cidr, 8, k + 128)]

Adjust NAT Gateway Configuration: Edit modules/vpc/main.tf:

enable_nat_gateway = true
single_nat_gateway = false  # Set to false for one NAT Gateway per AZ
one_nat_gateway_per_az = true

Security Group Rules

To customize security group rules:

Edit modules/ec2/main.tf to modify the security group:

# Add or modify ingress rules
ingress {
  from_port   = 9090
  to_port     = 9090
  protocol    = "tcp"
  cidr_blocks = var.allowed_cidr_blocks
  description = "Prometheus metrics"
}

Restrict egress traffic if needed:

# Replace the default egress rule
egress {
  from_port   = 443
  to_port     = 443
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
  description = "HTTPS outbound only"
}

Domain and DNS Configuration

To customize DNS settings:

Update terraform.tfvars:

domain_name = "your-custom-domain.com"
create_route53_records = true

Add additional DNS records by editing modules/route53/main.tf:

resource "aws_route53_record" "api" {
  zone_id = data.aws_route53_zone.selected.zone_id
  name    = "api.${var.domain_name}"
  type    = "A"
  ttl     = "300"
  records = [var.instance_public_ip]
}

Advanced Customization

High Availability Setup

To implement a high-availability setup:

Create an Auto Scaling Group Module
- Create modules/asg/main.tf, variables.tf, and outputs.tf
- Implement launch template and auto scaling group resources
- Configure health checks and scaling policies
Add a Load Balancer Module
- Create modules/alb/main.tf, variables.tf, and outputs.tf
- Configure listeners, target groups, and security groups
Update Main Configuration
- Replace the EC2 module with the ASG module in main.tf
- Add the ALB module to main.tf
- Update the Route53 module to point to the load balancer

HTTPS Support

To add HTTPS support:

Make sure enable_https is set to true in terraform.tfvars:
```
enable_https = true
```
Ensure that create_route53_records is set to true:
```
create_route53_records = true
```
Provide a valid email address for Let's Encrypt certificate notifications:
```
email_address = "your-email@example.com"
```
The deployment will automatically:
- Create a Let's Encrypt SSL certificate using Certbot
- Configure the API to use HTTPS
- Set up automatic certificate renewal
- Apply security headers for enhanced security
To customize the HTTPS configuration, modify the following files:
- modules/ec2/templates/user-data.sh.tpl: Certificate setup
- app/server.js: HTTPS server configuration

CI/CD Integration

To integrate with CI/CD pipelines:

Separate Build/Deploy Processes
- Move the Docker build logic out of Terraform
- Create CI/CD pipeline configuration files (GitHub Actions, Jenkins, etc.)
Add Terraform Remote Backend Configuration
- Update backend.tf to use a more collaborative backend like Terraform Cloud
Create Pipeline Stages
- Build and test the application
- Push images to ECR
- Run Terraform to update infrastructure
- Run integration tests

Customizing the vLLM Inference Engine

The solution comes with built-in support for the vLLM OpenAI-compatible API service. Here's how to customize it for your needs:

Changing the Model
- Update the model_id variable in terraform.tfvars to use a different model from HuggingFace
- Adjust max_model_len in terraform.tfvars to match the context window of the selected model
GPU Configuration
- Set use_gpu_instance to true in terraform.tfvars to use a GPU instance
- Modify gpu_instance_type to select a different instance type (e.g., g5.xlarge for newer GPUs)
- Adjust gpu_memory_utilization to control memory allocation (0.0-1.0)
Performance Tuning
- Modify the systemd service at /etc/systemd/system/vllm.service on the instance to add vLLM-specific parameters
- Common parameters include --tensor-parallel-size, --block-size, --swap-space, and --gpu-memory-utilization
Using Quantized Models
- For AWQ models, no special configuration is needed
- For GPTQ models, add --quantization gptq to the vLLM command line
- For GGUF models, a different approach may be required as vLLM doesn't natively support GGUF
Configuring HuggingFace Token
- The solution retrieves the HuggingFace token from SSM Parameter Store
- Create a secure string parameter in AWS SSM Parameter Store with the name specified in hf_token_parameter_name (default is /inference/hf_token)
- Store your HuggingFace token as the parameter value
- The instance has the necessary IAM permissions to retrieve this parameter
- For models that don't require a token, you can use a placeholder value

Example: Customizing for PyTorch Inference

Here's an example of customizing the solution for PyTorch inference:

Create a Custom Dockerfile

FROM pytorch/pytorch:1.12.0-cuda11.3-cudnn8-runtime

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY models/ /app/models/
COPY app/ /app/

EXPOSE 8080

CMD ["python", "server.py"]

Choose an Appropriate Instance Type

# In terraform.tfvars
instance_type = "g4dn.xlarge"  # GPU-enabled instance

Update IAM Permissions for S3

# In modules/ec2/main.tf
resource "aws_iam_policy" "s3_model_access" {
  name        = "${var.name}-s3-model-access"
  description = "Policy for S3 model access"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "s3:GetObject",
          "s3:ListBucket"
        ]
        Effect   = "Allow"
        Resource = [
          "arn:aws:s3:::${var.model_bucket}",
          "arn:aws:s3:::${var.model_bucket}/*"
        ]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "s3_model_access_attach" {
  role       = aws_iam_role.inference_instance.name
  policy_arn = aws_iam_policy.s3_model_access.arn
}

Update the systemd Service Template

# In modules/ec2/templates/inference-app.service.tpl
ExecStart=/usr/bin/docker run --rm --name inference-app \
    -p ${app_port}:${app_port} \
    -e PORT=${app_port} \
    -e AWS_REGION=${aws_region} \
    -e MODEL_BUCKET=${model_bucket} \
    --gpus all \
    ${ecr_repository_url}:latest

Navigation Menu

Uh oh!

AWS EC2 Inference Solution - Customization Guide

Customizing the Application

Modifying the API

Implementing a Different Backend

Customizing Infrastructure

Instance Type and Size

Networking Configuration

Security Group Rules

Domain and DNS Configuration

Advanced Customization

High Availability Setup

HTTPS Support

CI/CD Integration

Customizing the vLLM Inference Engine

Example: Customizing for PyTorch Inference

Uh oh!

FilesExpand file tree

CUSTOMIZATION.md

Latest commit

History

CUSTOMIZATION.md

File metadata and controls

AWS EC2 Inference Solution - Customization Guide

Customizing the Application

Modifying the API

Implementing a Different Backend

Customizing Infrastructure

Instance Type and Size

Networking Configuration

Security Group Rules

Domain and DNS Configuration

Advanced Customization

High Availability Setup

HTTPS Support

CI/CD Integration

Customizing the vLLM Inference Engine

Example: Customizing for PyTorch Inference