Devops (#35)

Locatelli-Flor · web-flow · commit 7291469dcc9f · 2025-12-17T19:55:09.000-03:00
* Adds initial Docker Compose setup

Sets up Docker Compose for local development.

Defines services for the API and database (PostgreSQL with pgvector).
Configures environment variables, volumes, and health checks for both services.
Also includes a Dockerfile that uses uv to manage the python environment and dependencies.

* Add initial database setup with tables for documents, document chunks, and chat sessions

* Refactor Docker setup: replace pgvector with new Dockerfiles for DocsManager and RAGManager, update docker-compose.yml for service configuration, and adjust Python version in RAGManager.

* Adds CI/CD workflows for deployment and validation

Sets up GitHub Actions workflows for continuous integration and continuous deployment.

- Introduces a deployment workflow that builds and pushes Docker images to ACR, configures kubectl, and restarts deployments in a Kubernetes namespace.
- Implements a pull request validation workflow that performs secret scanning with Gitleaks, builds Docker images for validation (without pushing), runs Trivy vulnerability scans, and uploads the results to GitHub Security.
- Adds a PR summary workflow that posts a comment on the pull request with the results of the Gitleaks and build validation jobs, including a notice to check the security tab for any found vulnerabilities.

* Refactors PR validation workflow for clarity

Streamlines the PR validation workflow by removing the Gitleaks job and improving the presentation of Trivy results.

The workflow now focuses on build validation and vulnerability scanning with clearer output in the PR summary. Trivy results are now displayed in a table format within the PR comment, and a direct link to the detailed results in the Actions tab is included. The Gitleaks check is removed.

* Enhances deployment workflow with summaries

Adds deployment summary to the workflow, providing detailed information about the deployed service, image, and pod status in the job summary.

Also, it includes a success notification with links to deployed services and sets fail-fast to false to ensure all services are deployed.

* Enhances deployment workflow with rollback

Improves the deployment process by adding rollback capabilities on failure, enhanced logging, and deployment summaries in GitHub.
The changes also include updating the deployment strategy from rolling restarts to image updates.
It adds timeout configurations for deployments.
Also adds live URL information to success summary.

* Add GitHub Actions workflows for deployment and PR validation

* Refactor Dockerfiles to streamline installation and add healthcheck

* Refactor Dockerfiles to simplify file copying and improve build process

* Fix CMD syntax in Dockerfiles for consistency

* Fix CMD syntax in Dockerfiles for consistency

* Update Python version to 3.13 and fix CMD syntax in Dockerfiles for consistency

* Updates project dependencies

Updates the project's dependencies in pyproject.toml
to align with the current versions and includes
'langchain-text-splitters' to resolve the name change of
'langchain.text_splitter'.
Adds a start script to `pyproject.toml`.
Updates the poetry lock file.

* Refactor import statement for clarity and add pdfplumber and minio dependencies

* Remove rollback step from deployment workflow and adjust script section in pyproject.toml for clarity

* Replace rollout status with wait for new pods to be ready in deployment workflow

* Increase deployment timeout to 8 minutes and update rollout status check in deployment workflow

* Remove redundant wait command from deployment rollout step in workflow

* Refactor deployment workflow to improve rollout checks and update timeout settings

* Add Discord notifications for successful and failed deployments, and new pull requests

* Enhance Discord notifications with detailed deployment information for success and failure events

* Update PR validation trigger to include opened event and adjust Python version requirement in uv.lock

* Reduce rollout timeout to 30s and add debug steps for deployment status before and after rollout

* Update rollout timeout to 60s and remove debug steps from deployment workflow

* Add SonarQube scan step to PR validation workflow

* Refactor configuration settings to use Pydantic Field for environment variables

* Add env_map for PostgreSQL configuration in Settings

* Refactor PostgreSQL and RabbitMQ configuration to remove env_map and use Field for environment variables

* Update Dockerfile to include Python in CMD and improve uv sync command
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -9,10 +9,14 @@ on:
 env:
   REGISTRY: crretoxmas2024.azurecr.io
   NAMESPACE: reto-xmas-2025-goland-ia-backend
+  ROLLOUT_TIMEOUT: 60s
+  READY_CHECK_RETRIES: 20
+  READY_CHECK_SLEEP: 15
 
 jobs:
   build-and-deploy:
     runs-on: ubuntu-latest
+    timeout-minutes: 15
     strategy:
       fail-fast: false
       matrix:
@@ -25,7 +29,7 @@ jobs:
             path: ./RAGManager
             image: reto-xmas-2025-goland-ia-backend-rag-manager
             deployment: rag-manager
-    
+
     steps:
     - name: Checkout code
       uses: actions/checkout@v4
@@ -51,6 +55,7 @@ jobs:
           ${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}
         cache-from: type=registry,ref=${{ env.REGISTRY }}/${{ matrix.service.image }}:buildcache
         cache-to: type=registry,ref=${{ env.REGISTRY }}/${{ matrix.service.image }}:buildcache,mode=max
+        provenance: false
 
     - name: Set up kubectl
       uses: azure/setup-kubectl@v3
@@ -63,28 +68,78 @@ jobs:
         echo "${{ secrets.KUBECONFIG }}" | base64 -d > $HOME/.kube/config
         chmod 600 $HOME/.kube/config
 
-    - name: Restart deployment
+    - name: Update deployment image
       run: |
-        kubectl rollout restart deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }}
-        kubectl rollout status deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }} --timeout=5m
+        kubectl set image deployment/${{ matrix.service.deployment }} \
+          api=${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }} \
+          -n ${{ env.NAMESPACE }}
 
-    - name: Verify deployment
+    - name: Wait for deployment to be ready (robust)
       run: |
-        echo "✅ Deployment successful for ${{ matrix.service.name }}"
-        kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }}
+        set -e
+
+        echo "Checking rollout status (non-blocking)..."
+        kubectl rollout status deployment/${{ matrix.service.deployment }} \
+          -n ${{ env.NAMESPACE }} \
+          --timeout=${{ env.ROLLOUT_TIMEOUT }} || true
+
+        echo "Waiting for available replicas..."
+
+        for i in $(seq 1 $READY_CHECK_RETRIES); do
+          DESIRED=$(kubectl get deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }} -o jsonpath='{.spec.replicas}')
+          AVAILABLE=$(kubectl get deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }} -o jsonpath='{.status.availableReplicas}')
+
+          echo "Attempt $i: $AVAILABLE / $DESIRED replicas available"
+
+          if [ "$AVAILABLE" = "$DESIRED" ]; then
+            echo "Deployment is ready"
+            exit 0
+          fi
+
+          sleep $READY_CHECK_SLEEP
+        done
+
+        echo "Deployment did not become ready in time"
+        kubectl describe deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }}
+        exit 1
+
+    - name: Verify pods
+      if: success()
+      run: |
+        kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }} -o wide
+
+    - name: Get logs on failure
+      if: failure()
+      run: |
+        echo "=== Deployment ==="
+        kubectl describe deployment/${{ matrix.service.deployment }} -n ${{ env.NAMESPACE }}
+
+        echo "=== Pods ==="
+        kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }} -o wide
+
+        echo "=== Pod Logs ==="
+        kubectl logs -n ${{ env.NAMESPACE }} \
+          -l app=${{ matrix.service.deployment }} \
+          --tail=100 \
+          --all-containers=true \
+          --prefix=true || true
 
     - name: Deployment Summary
       if: always()
       run: |
-        echo "### 🚀 Deployment Summary" >> $GITHUB_STEP_SUMMARY
+        STATUS="${{ job.status }}"
+
+        echo "### Deployment - ${{ matrix.service.name }}" >> $GITHUB_STEP_SUMMARY
         echo "" >> $GITHUB_STEP_SUMMARY
-        echo "**Service:** ${{ matrix.service.name }}" >> $GITHUB_STEP_SUMMARY
-        echo "**Image:** ${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}" >> $GITHUB_STEP_SUMMARY
-        echo "**Status:** ${{ job.status }}" >> $GITHUB_STEP_SUMMARY
+        echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
+        echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
+        echo "| Image | \`${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}\` |" >> $GITHUB_STEP_SUMMARY
+        echo "| Status | $STATUS |" >> $GITHUB_STEP_SUMMARY
         echo "" >> $GITHUB_STEP_SUMMARY
+
         echo "#### Pods:" >> $GITHUB_STEP_SUMMARY
         echo '```' >> $GITHUB_STEP_SUMMARY
-        kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }} >> $GITHUB_STEP_SUMMARY
+        kubectl get pods -n ${{ env.NAMESPACE }} -l app=${{ matrix.service.deployment }} >> $GITHUB_STEP_SUMMARY || true
         echo '```' >> $GITHUB_STEP_SUMMARY
 
   notify-success:
@@ -95,8 +150,34 @@ jobs:
     steps:
     - name: Success Summary
       run: |
-        echo "### ✅ Deployment Successful!" >> $GITHUB_STEP_SUMMARY
+        echo "### All Services Deployed" >> $GITHUB_STEP_SUMMARY
         echo "" >> $GITHUB_STEP_SUMMARY
-        echo "All services deployed successfully:" >> $GITHUB_STEP_SUMMARY
-        echo "- 🌐 DocsManager: https://goland-ia-backend-docs-manager.reto-ucu.net" >> $GITHUB_STEP_SUMMARY
-        echo "- 🌐 RAGManager: https://goland-ia-backend-rag-manager.reto-ucu.net" >> $GITHUB_STEP_SUMMARY
+        echo "**Live URLs:**" >> $GITHUB_STEP_SUMMARY
+        echo "- [DocsManager](https://goland-ia-backend-docs-manager.reto-ucu.net/docs)" >> $GITHUB_STEP_SUMMARY
+        echo "- [RAGManager](https://goland-ia-backend-rag-manager.reto-ucu.net/docs)" >> $GITHUB_STEP_SUMMARY
+
+    - name: Notificar a Discord (deploy exitoso)
+      run: |
+        curl -H "Content-Type: application/json" \
+          -X POST \
+          -d "{\"content\": \"✅ **Deploy exitoso**\\n\\n**Servicio:** \\\\`${{ matrix.service.name }}\\\\`\\n**Imagen:** \\\\`${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}\\\\`\\n**Namespace:** \\\\`${{ env.NAMESPACE }}\\\\`\\n**Autor:** ${{ github.actor }}\\n**Commit:** [${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }})\\n\\n[Ver ejecución en GitHub Actions](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\\n\\n:rocket: ¡Todo salió bien!\"}" \
+          https://discord.com/api/webhooks/1449147972330852463/sMs3QYOvQ6oiaPHR4PlBluwNrhWSsVmfm3_Miz6UAFrPxj1SsbqNnHXc5eK9h8ZSaumk
+
+  notify-failure:
+    name: Deployment Failed
+    runs-on: ubuntu-latest
+    needs: [build-and-deploy]
+    if: failure()
+    steps:
+    - name: Failure Summary
+      run: |
+        echo "### Deployment Failed" >> $GITHUB_STEP_SUMMARY
+        echo "" >> $GITHUB_STEP_SUMMARY
+        echo "Please check deployment logs above." >> $GITHUB_STEP_SUMMARY
+
+    - name: Notificar a Discord (deploy fallido)
+      run: |
+        curl -H "Content-Type: application/json" \
+          -X POST \
+          -d "{\"content\": \"❌ **Falló el deploy**\\n\\n**Servicio:** \\\\`${{ matrix.service.name }}\\\\`\\n**Imagen:** \\\\`${{ env.REGISTRY }}/${{ matrix.service.image }}:${{ github.sha }}\\\\`\\n**Namespace:** \\\\`${{ env.NAMESPACE }}\\\\`\\n**Autor:** ${{ github.actor }}\\n**Commit:** [${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }})\\n\\n[Ver ejecución en GitHub Actions](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})\\n\\n:warning: Revisa los logs para más detalles.\"}" \
+          https://discord.com/api/webhooks/1449147972330852463/sMs3QYOvQ6oiaPHR4PlBluwNrhWSsVmfm3_Miz6UAFrPxj1SsbqNnHXc5eK9h8ZSaumk
diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml
@@ -2,6 +2,7 @@ name: PR Validation
 
 on:
   pull_request:
+    types: [opened]
     branches:
       - main
 
@@ -66,6 +67,13 @@ jobs:
         severity: 'CRITICAL,HIGH'
         exit-code: '0'
 
+    - name: SonarQube Scan
+      uses: sonarsource/sonarqube-scan-action@v2
+      with:
+        host: https://sonarqube.reto-ucu.net/
+        login: ${{ secrets.SONAR_TOKEN }}
+        projectKey: reto-xmas-2025-goland-ia-backend
+
   pr-summary:
     name: PR Summary
     runs-on: ubuntu-latest
@@ -96,4 +104,16 @@ jobs:
             owner: context.repo.owner,
             repo: context.repo.repo,
             body: message
-          });
+          });
+
+  discord-pr-notify:
+    name: Notificar PR en Discord
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+    steps:
+      - name: Notificar a Discord (nuevo PR)
+        run: |
+          curl -H "Content-Type: application/json" \
+            -X POST \
+            -d "{\"content\": \"🔔 Nuevo Pull Request: [${{ github.event.pull_request.title }}](${{ github.event.pull_request.html_url }}) por ${{ github.event.pull_request.user.login }}\"}" \
+            https://discord.com/api/webhooks/1449147972330852463/sMs3QYOvQ6oiaPHR4PlBluwNrhWSsVmfm3_Miz6UAFrPxj1SsbqNnHXc5eK9h8ZSaumk
diff --git a/DocsManager/.python-version b/DocsManager/.python-version
@@ -1 +1 @@
-3.14
+3.13
diff --git a/DocsManager/Dockerfile b/DocsManager/Dockerfile
@@ -1,13 +1,26 @@
-FROM ghcr.io/astral-sh/uv:python3.13-bookworm-slim
+FROM python:3.12-slim
 
 WORKDIR /app
 
-COPY pyproject.toml uv.lock* ./
+RUN apt-get update && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
+
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
+    mv /root/.local/bin/uv /usr/local/bin/uv && \
+    mv /root/.local/bin/uvx /usr/local/bin/uvx
+
 
-RUN uv sync --frozen --no-cache || uv sync --no-cache
 
 COPY . .
 
+RUN uv sync --no-dev --no-cache \
+    && uv pip list \
+    && uv pip show fastapi uvicorn
+
 EXPOSE 8000
 
-CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+CMD ["uv", "run", "--no-sync", "python", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
diff --git a/DocsManager/app/core/config.py b/DocsManager/app/core/config.py
@@ -1,5 +1,5 @@
 from pydantic_settings import BaseSettings, SettingsConfigDict
-from pydantic import field_validator, model_validator
+from pydantic import field_validator, model_validator, Field
 from typing import Optional
 from urllib.parse import quote_plus
 import logging
@@ -29,7 +29,6 @@ class Settings(BaseSettings):
     minio_access_key: str
     minio_secret_key: str
     minio_bucket: str = "documents"
-    minio_folder: str = "rag-docs"
     minio_use_ssl: bool = True
 
     # Database Configuration (for SQLAlchemy)
diff --git a/DocsManager/pyproject.toml b/DocsManager/pyproject.toml
@@ -3,10 +3,14 @@ name = "goland-ia"
 version = "0.1.0"
 description = "Add your description here"
 readme = "README.md"
-requires-python = ">=3.14"
+requires-python = ">=3.12,<3.14"
+
 dependencies = [
     "fastapi>=0.124.2",
-    "ipython>=9.8.0",
+    "sqlalchemy>=2.0.35",
+    "asyncpg>=0.29.0",
+    "psycopg2-binary>=2.9.9",
+    "langgraph>=1.0.4",
     "typing-extensions>=4.15.0",
     "uvicorn>=0.38.0",
     "sqlalchemy>=2.0.0",
@@ -18,14 +22,17 @@ dependencies = [
     "python-multipart>=0.0.6",
 ]
 
+[project.scripts]
+start = "uvicorn main:app --reload --host 0.0.0.0 --port 8000"
+
 [project.optional-dependencies]
 dev = [
     "ruff>=0.14.0",
 ]
 
 [tool.ruff]
 # Enable pyproject.toml support
-target-version = "py314"
+target-version = "py313"
 line-length = 120
 
 # Enable auto-fix
diff --git a/DocsManager/uv.lock b/DocsManager/uv.lock
diff --git a/RAGManager/Dockerfile b/RAGManager/Dockerfile
@@ -1,13 +1,22 @@
-FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
+FROM python:3.12-slim
 
 WORKDIR /app
 
-COPY pyproject.toml uv.lock* ./
+RUN apt-get update && apt-get install -y \
+    curl \
+    && rm -rf /var/lib/apt/lists/*
 
-RUN uv sync --frozen --no-cache || uv sync --no-cache
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
+    mv /root/.local/bin/uv /usr/local/bin/uv && \
+    mv /root/.local/bin/uvx /usr/local/bin/uvx
 
 COPY . .
 
+RUN uv sync --no-dev --no-cache
+
 EXPOSE 8000
 
-CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
+HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
+    CMD curl -f http://localhost:8000/health || exit 1
+
+CMD ["uv", "run", "--no-sync", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
diff --git a/RAGManager/pyproject.toml b/RAGManager/pyproject.toml
