Skip to content

Commit b1e20d4

Browse files
Locatelli-FlorLocatelli-Flor
authored
Devops cositas (#5)
* Adds initial Docker Compose setup Sets up Docker Compose for local development. Defines services for the API and database (PostgreSQL with pgvector). Configures environment variables, volumes, and health checks for both services. Also includes a Dockerfile that uses uv to manage the python environment and dependencies. * Add initial database setup with tables for documents, document chunks, and chat sessions * Refactor Docker setup: replace pgvector with new Dockerfiles for DocsManager and RAGManager, update docker-compose.yml for service configuration, and adjust Python version in RAGManager. * Adds CI/CD workflows for deployment and validation Sets up GitHub Actions workflows for continuous integration and continuous deployment. - Introduces a deployment workflow that builds and pushes Docker images to ACR, configures kubectl, and restarts deployments in a Kubernetes namespace. - Implements a pull request validation workflow that performs secret scanning with Gitleaks, builds Docker images for validation (without pushing), runs Trivy vulnerability scans, and uploads the results to GitHub Security. - Adds a PR summary workflow that posts a comment on the pull request with the results of the Gitleaks and build validation jobs, including a notice to check the security tab for any found vulnerabilities. * Refactors PR validation workflow for clarity Streamlines the PR validation workflow by removing the Gitleaks job and improving the presentation of Trivy results. The workflow now focuses on build validation and vulnerability scanning with clearer output in the PR summary. Trivy results are now displayed in a table format within the PR comment, and a direct link to the detailed results in the Actions tab is included. The Gitleaks check is removed. * Enhances deployment workflow with summaries Adds deployment summary to the workflow, providing detailed information about the deployed service, image, and pod status in the job summary. Also, it includes a success notification with links to deployed services and sets fail-fast to false to ensure all services are deployed.
1 parent 6a5b4ad commit b1e20d4

File tree

11 files changed

+379
-53
lines changed

11 files changed

+379
-53
lines changed

‎.github/workflows/deploy.yml‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
name: Deploy to Kubernetes
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
workflow_dispatch:
8+
9+
env:
10+
REGISTRY: crretoxmas2024.azurecr.io
11+
NAMESPACE: reto-xmas-2025-goland-ia-backend
12+
13+
jobs:
14+
build-and-deploy:
15+
runs-on: ubuntu-latest
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
service:
20+
- name: docs-manager
21+
path: ./DocsManager
22+
image: reto-xmas-2025-goland-ia-backend-docs-manager
23+
deployment: docs-manager
24+
- name: rag-manager
25+
path: ./RAGManager
26+
image: reto-xmas-2025-goland-ia-backend-rag-manager
27+
deployment: rag-manager
28+
29+
steps:
30+
- name: Checkout code
31+
uses: actions/checkout@v4
32+
33+
- name: Set up Docker Buildx
34+
uses: docker/setup-buildx-action@v3
35+
36+
- name: Login to ACR
37+
uses: docker/login-action@v3
38+
with:
39+
registry: ${{ env.REGISTRY }}
40+
username: ${{ secrets.ACR_USERNAME }}
41+
password: ${{ secrets.ACR_PASSWORD }}
42+
43+
- name: Build and push Docker image
44+
uses: docker/build-push-action@v5
45+
with:
46+
context: ${{ matrix.service.path }}
47+
platforms: linux/amd64
48+
push: true
49+
tags: |
50+
${{ env.REGISTRY }}/${{ matrix.service.image }}:latest
51+
${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}
52+
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ matrix.service.image }}:buildcache
53+
cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ matrix.service.image }}:buildcache,mode=max
54+
55+
- name: Set up kubectl
56+
uses: azure/setup-kubectl@v3
57+
with:
58+
version: 'latest'
59+
60+
- name: Configure kubectl
61+
run: |
62+
mkdir -p $HOME/.kube
63+
echo "${{ secrets.KUBECONFIG }}" | base64 -d > $HOME/.kube/config
64+
chmod 600 $HOME/.kube/config
65+
66+
- name: Restart deployment
67+
run: |
68+
kubectl rollout restart deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }}
69+
kubectl rollout status deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }} --timeout=5m
70+
71+
- name: Verify deployment
72+
run: |
73+
echo "✅ Deployment successful for ${{ matrix.service.name }}"
74+
kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }}
75+
76+
- name: Deployment Summary
77+
if: always()
78+
run: |
79+
echo "### 🚀 Deployment Summary" >> $GITHUB_STEP_SUMMARY
80+
echo "" >> $GITHUB_STEP_SUMMARY
81+
echo "**Service:** ${{ matrix.service.name }}" >> $GITHUB_STEP_SUMMARY
82+
echo "**Image:** ${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}" >> $GITHUB_STEP_SUMMARY
83+
echo "**Status:** ${{ job.status }}" >> $GITHUB_STEP_SUMMARY
84+
echo "" >> $GITHUB_STEP_SUMMARY
85+
echo "#### Pods:" >> $GITHUB_STEP_SUMMARY
86+
echo '```' >> $GITHUB_STEP_SUMMARY
87+
kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }} >> $GITHUB_STEP_SUMMARY
88+
echo '```' >> $GITHUB_STEP_SUMMARY
89+
90+
notify-success:
91+
name: Deployment Success
92+
runs-on: ubuntu-latest
93+
needs: [build-and-deploy]
94+
if: success()
95+
steps:
96+
- name: Success Summary
97+
run: |
98+
echo "### ✅ Deployment Successful!" >> $GITHUB_STEP_SUMMARY
99+
echo "" >> $GITHUB_STEP_SUMMARY
100+
echo "All services deployed successfully:" >> $GITHUB_STEP_SUMMARY
101+
echo "- 🌐 DocsManager: https://goland-ia-backend-docs-manager.reto-ucu.net" >> $GITHUB_STEP_SUMMARY
102+
echo "- 🌐 RAGManager: https://goland-ia-backend-rag-manager.reto-ucu.net" >> $GITHUB_STEP_SUMMARY

‎.github/workflows/pr-validation.yml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
name: PR Validation
2+
3+
on:
4+
pull_request:
5+
branches:
6+
- main
7+
8+
env:
9+
REGISTRY: crretoxmas2024.azurecr.io
10+
DOCS_MANAGER_IMAGE: reto-xmas-2025-goland-ia-backend-docs-manager
11+
RAG_MANAGER_IMAGE: reto-xmas-2025-goland-ia-backend-rag-manager
12+
13+
jobs:
14+
build-validation:
15+
name: Build Validation
16+
runs-on: ubuntu-latest
17+
strategy:
18+
fail-fast: false
19+
matrix:
20+
service:
21+
- name: docs-manager
22+
path: ./DocsManager
23+
image: reto-xmas-2025-goland-ia-backend-docs-manager
24+
- name: rag-manager
25+
path: ./RAGManager
26+
image: reto-xmas-2025-goland-ia-backend-rag-manager
27+
28+
steps:
29+
- name: Checkout code
30+
uses: actions/checkout@v4
31+
32+
- name: Set up Docker Buildx
33+
uses: docker/setup-buildx-action@v3
34+
35+
- name: Build Docker image
36+
uses: docker/build-push-action@v5
37+
with:
38+
context: ${{ matrix.service.path }}
39+
platforms: linux/amd64
40+
load: true
41+
tags: ${{ matrix.service.image }}:pr-${{ github.event.pull_request.number }}
42+
cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ matrix.service.image }}:buildcache
43+
44+
- name: Run Trivy vulnerability scanner
45+
uses: aquasecurity/trivy-action@master
46+
with:
47+
image-ref: ${{ matrix.service.image }}:pr-${{ github.event.pull_request.number }}
48+
format: 'sarif'
49+
output: 'trivy-results-${{ matrix.service.name }}.sarif'
50+
severity: 'CRITICAL,HIGH'
51+
exit-code: '0'
52+
53+
- name: Upload Trivy results to GitHub Security
54+
uses: github/codeql-action/upload-sarif@v4
55+
if: always()
56+
with:
57+
sarif_file: 'trivy-results-${{ matrix.service.name }}.sarif'
58+
category: 'trivy-${{ matrix.service.name }}'
59+
60+
- name: Print Trivy results
61+
if: always()
62+
uses: aquasecurity/trivy-action@master
63+
with:
64+
image-ref: ${{ matrix.service.image }}:pr-${{ github.event.pull_request.number }}
65+
format: 'table'
66+
severity: 'CRITICAL,HIGH'
67+
exit-code: '0'
68+
69+
pr-summary:
70+
name: PR Summary
71+
runs-on: ubuntu-latest
72+
needs: [build-validation]
73+
if: always()
74+
steps:
75+
- name: PR Comment
76+
uses: actions/github-script@v7
77+
with:
78+
script: |
79+
const buildStatus = '${{ needs.build-validation.result }}';
80+
81+
const statusEmoji = (status) => {
82+
if (status === 'success') return '✅';
83+
if (status === 'failure') return '❌';
84+
return '⚠️';
85+
};
86+
87+
let message = '## 🔍 PR Validation Results\n\n';
88+
message += `| Check | Status |\n`;
89+
message += `|-------|--------|\n`;
90+
message += `| Build | ${statusEmoji(buildStatus)} ${buildStatus} |\n`;
91+
message += `| Trivy | Check Security tab |\n\n`;
92+
message += `[View detailed results](${context.payload.repository.html_url}/actions/runs/${context.runId})`;
93+
94+
github.rest.issues.createComment({
95+
issue_number: context.issue.number,
96+
owner: context.repo.owner,
97+
repo: context.repo.repo,
98+
body: message
99+
});

‎DocsManager/Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim
2+
3+
WORKDIR /app
4+
5+
COPY pyproject.toml uv.lock* ./
6+
7+
RUN uv sync --frozen --no-cache || uv sync --no-cache
8+
9+
COPY . .
10+
11+
EXPOSE 8000
12+
13+
CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

‎DocsManager/Dockerfile.pgvector‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎DocsManager/docker-compose.yml‎

Lines changed: 0 additions & 23 deletions
This file was deleted.

‎RAGManager/.python-version‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
3.14
1+
3.12

‎RAGManager/Dockerfile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
2+
3+
WORKDIR /app
4+
5+
COPY pyproject.toml uv.lock* ./
6+
7+
RUN uv sync --frozen --no-cache || uv sync --no-cache
8+
9+
COPY . .
10+
11+
EXPOSE 8000
12+
13+
CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

‎RAGManager/Dockerfile.pgvector‎

Lines changed: 0 additions & 5 deletions
This file was deleted.

‎RAGManager/docker-compose.yml‎

Lines changed: 0 additions & 23 deletions
This file was deleted.

‎compose.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
services:
2+
docs-manager:
3+
build:
4+
context: ./DocsManager
5+
dockerfile: Dockerfile
6+
container_name: docs-manager
7+
ports:
8+
- "8000:8000"
9+
env_file:
10+
- .env
11+
- ./DocsManager/.env
12+
depends_on:
13+
db:
14+
condition: service_healthy
15+
restart: unless-stopped
16+
environment:
17+
- PYTHONUNBUFFERED=1
18+
- SERVICE_NAME=docs-manager
19+
- SERVICE_ROLE=document-handler
20+
21+
rag-manager:
22+
build:
23+
context: ./RAGManager
24+
dockerfile: Dockerfile
25+
container_name: rag-manager
26+
ports:
27+
- "8001:8000"
28+
env_file:
29+
- .env
30+
- ./RAGManager/.env
31+
depends_on:
32+
db:
33+
condition: service_healthy
34+
rabbitmq:
35+
condition: service_healthy
36+
restart: unless-stopped
37+
environment:
38+
- PYTHONUNBUFFERED=1
39+
- SERVICE_NAME=rag-manager
40+
- SERVICE_ROLE=document-processor
41+
42+
db:
43+
image: pgvector/pgvector:pg16
44+
container_name: postgres-db
45+
env_file:
46+
- .env
47+
environment:
48+
POSTGRES_USER: ${POSTGRES_USER}
49+
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
50+
POSTGRES_DB: ${POSTGRES_DB}
51+
ports:
52+
- "5432:5432"
53+
volumes:
54+
- postgres_data:/var/lib/postgresql/data
55+
- ./db-init:/docker-entrypoint-initdb.d
56+
restart: unless-stopped
57+
healthcheck:
58+
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
59+
interval: 5s
60+
timeout: 5s
61+
retries: 5
62+
63+
rabbitmq:
64+
image: rabbitmq:3.13-management-alpine
65+
container_name: rabbitmq
66+
ports:
67+
- "5672:5672"
68+
- "15672:15672"
69+
env_file:
70+
- .env
71+
environment:
72+
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
73+
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
74+
volumes:
75+
- rabbitmq_data:/var/lib/rabbitmq
76+
restart: unless-stopped
77+
healthcheck:
78+
test: ["CMD", "rabbitmq-diagnostics", "ping"]
79+
interval: 10s
80+
timeout: 5s
81+
retries: 5
82+
83+
volumes:
84+
postgres_data:
85+
rabbitmq_data:

0 commit comments

Comments
 (0)