Mixpeek Pipeline CI/CD with GitHub Actions

This repository demonstrates how to automatically build and push Docker containers to AWS ECS for each subdirectory in the repository using GitHub Actions.

Directory Structure

The repository should be structured as follows:

my-repo/
├── .github/
│ └── workflows/
│ └── build-and-push-docker.yml
├── pipeline1/
│ ├── handler.py
│ └── requirements.txt
├── pipeline2/
│ ├── handler.py
│ └── requirements.txt
└── pipeline3/
├── handler.py
└── requirements.txt

• .github/workflows/build-and-push-docker.yml: The GitHub Actions workflow file.
• pipeline1, pipeline2, pipeline3: Example services. Each service contains a handler.py file and a requirements.txt file.

Prerequisites

1. AWS Account: You need an AWS account and an ECR repository set up.
2. GitHub Secrets: Configure the following secrets in your GitHub repository settings:
   • AWS_ACCOUNT_ID: Your AWS account ID.
   • AWS_ACCESS_KEY_ID: Your AWS access key ID.
   • AWS_SECRET_ACCESS_KEY: Your AWS secret access key.

Setting Up the Workflow

1. Clone the Repository:

   git clone https://github.com/your-username/your-repo.git
   cd your-repo

2. Create the GitHub Actions Workflow:

   • Create a .github/workflows directory if it doesn't exist.
   • Create a file named build-and-push-docker.yml inside .github/workflows/ with the following content:

   name: Build and Push Docker Containers
   
   on:
   push:
       paths:
       - "**/handler.py"
       - "**/requirements.txt"
   
   jobs:
   build-and-push:
       runs-on: ubuntu-latest
   
       env:
       AWS_REGION: us-east-1
       ECR_REPOSITORY: your-ecr-repository # Replace with your ECR repository
       AWS_ACCOUNT_ID: ${{ secrets.AWS_ACCOUNT_ID }}
       AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
       AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
   
       steps:
       - name: Checkout code
           uses: actions/checkout@v2
   
       - name: Configure AWS credentials
           uses: aws-actions/configure-aws-credentials@v1
           with:
           aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
           aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
           aws-region: $AWS_REGION
   
       - name: Build and push Docker images
           run: |
           for dir in $(find . -type d -maxdepth 1 -mindepth 1); do
               if [[ -f "$dir/handler.py" && -f "$dir/requirements.txt" ]]; then
               foldername=$(basename $dir)
               index_id=$(git rev-parse --short HEAD)
               cat > $dir/Dockerfile <<EOF
               FROM python:3.10-slim
               WORKDIR /app
               COPY requirements.txt .
               RUN pip install --no-cache-dir -r requirements.txt
               COPY . .
               CMD ["python", "handler.py"]
               EOF
               docker build -t $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$foldername-$index_id:latest $dir
               docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$ECR_REPOSITORY:$foldername-$index_id:latest
               fi
           done
   
       - name: Log out from Docker Hub
           run: docker logout

3. Add Your Service Directories:

   • Create directories for each service (e.g., service1, service2, etc.).

   • Each directory should contain:

     • handler.py: Your Python code.
     • requirements.txt: Your Python dependencies.

     Example handler.py:

     def main():
         print("Hello from service1!")
     
     if __name__ == "__main__":
         main()

     Example requirements.txt:

     # List your Python dependencies here
     requests
     

4. Commit and Push:

   git add .
   git commit -m "Set up Docker CI/CD workflow"
   git push origin main

How It Works

• The GitHub Actions workflow is triggered on changes to any handler.py or requirements.txt file.
• For each subdirectory containing a handler.py and requirements.txt, a Docker image is built and pushed to your AWS ECR repository.
• The Docker images are tagged with the folder name and the latest commit hash.

Notes

• Ensure your AWS ECR repository exists and you have the correct permissions to push images.
• Adjust the AWS_REGION and ECR_REPOSITORY in the workflow file as needed.

That's it! Your services will now be automatically built and pushed to AWS ECS whenever there are changes.