Skip to content

Commit 80486eb

Browse files
diraoldiraol
authored
Habilita CI/CD multi-arquitetura e melhora infraestrutura de testes (#106)
## Resumo Este PR reativa e melhora significativamente a infraestrutura de CI/CD do projeto, adicionando suporte completo para múltiplas arquiteturas (AMD64 e ARM64) e ferramentas aprimoradas de depuração de testes. ## Mudanças Principais ### 🚀 CI/CD - Reativa workflows do GitHub Actions (test_pull_request e lint_and_unit) - Adiciona suporte completo para builds multi-arquitetura (linux/amd64 e linux/arm64) - Workflows agora constroem e testam nativamente em ambas as arquiteturas - Cria manifestos multi-arquitetura automaticamente - Publica imagens para GitHub Container Registry (ghcr.io) ### 🏗️ Infraestrutura - Adiciona scripts de inicialização para PostgreSQL e OpenSearch - Atualiza MinIO para versão compatível com ARM64 - Melhora docker-compose.yml com healthchecks e inicialização automatizada - Corrige compatibilidade de comandos no Makefile ### 🧪 Testes - Adiciona test runner instrumentado com logging detalhado e timing - Corrige suíte de testes para corresponder à implementação atual - Melhora visibilidade de execução de testes no CI ### 🐳 Docker - Atualiza Dockerfile do Apache Tika para multi-arquitetura - Otimiza builds com cache do GitHub Actions - Testes abrangentes em ambas as arquiteturas ## Benefícios ✅ **Suporte ARM64**: Projeto agora roda nativamente em Macs Apple Silicon e instâncias cloud ARM ✅ **CI Confiável**: Testes automatizados para todas as PRs e commits na main ✅ **Melhor Depuração**: Test runner instrumentado identifica rapidamente problemas ✅ **Ambiente Consistente**: Inicialização automatizada de serviços garante ambiente reproduzível ✅ **Builds Rápidos**: Cache otimizado reduz tempo de build ## Testes - ✅ Builds multi-arquitetura testados - ✅ Testes unitários passando em ambas as arquiteturas - ✅ Serviços (PostgreSQL, OpenSearch, MinIO) testados ## Compatibilidade Este PR mantém total compatibilidade com: - Ambientes de desenvolvimento existentes - Deployments AMD64 atuais - Adiciona suporte para ARM64 sem breaking changes
2 parents ac56a3c + b14b6d9 commit 80486eb

15 files changed

Lines changed: 745 additions & 295 deletions

‎.github/workflows/build_apache_tika.yaml‎

Lines changed: 0 additions & 71 deletions
This file was deleted.

.github/workflows_disabled/build_container_image.yaml renamed to .github/workflows/build_container_image.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ jobs:
190190
- 9200:9200
191191

192192
minio:
193-
image: bitnami/minio:2021.4.6
193+
image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
194194
env:
195195
MINIO_ACCESS_KEY: minio-access-key
196196
MINIO_SECRET_KEY: minio-secret-key

‎.github/workflows/lint_and_unit.yaml‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
name: Lint and unit checks
2+
3+
on:
4+
pull_request:
5+
branches: [ main ]
6+
push:
7+
branches: [ main ]
8+
9+
permissions:
10+
contents: read
11+
12+
jobs:
13+
lint:
14+
name: Python lint with pre-commit
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
- uses: actions/setup-python@v5
19+
with:
20+
python-version: '3.11'
21+
cache: 'pip'
22+
cache-dependency-path: |
23+
requirements-dev.txt
24+
requirements.txt
25+
- name: Install dependencies
26+
run: |
27+
python -m pip install --upgrade pip
28+
pip install -r requirements-dev.txt
29+
- name: Run pre-commit hooks
30+
run: |
31+
pre-commit run --all-files
32+
33+
unit:
34+
name: Unit tests (${{ matrix.arch }})
35+
runs-on: ${{ matrix.runner }}
36+
needs: lint
37+
strategy:
38+
matrix:
39+
include:
40+
- arch: amd64
41+
runner: ubuntu-latest
42+
platform: linux/amd64
43+
- arch: arm64
44+
runner: ubuntu-24.04-arm
45+
platform: linux/arm64
46+
env:
47+
PLATFORM: ${{ matrix.platform }}
48+
services:
49+
postgres:
50+
image: postgres:11
51+
env:
52+
POSTGRES_PASSWORD: queridodiario
53+
POSTGRES_USER: queridodiario
54+
POSTGRES_DB: queridodiariodb
55+
options: >-
56+
--health-cmd pg_isready
57+
--health-interval 10s
58+
--health-timeout 5s
59+
--health-retries 5
60+
ports:
61+
- 5432:5432
62+
63+
opensearch:
64+
image: opensearchproject/opensearch:2.9.0
65+
env:
66+
discovery.type: single-node
67+
plugins.security.ssl.http.enabled: false
68+
plugins.security.disabled: true
69+
OPENSEARCH_INITIAL_ADMIN_PASSWORD: admin
70+
options: >-
71+
--health-cmd "curl -f http://localhost:9200/_cluster/health"
72+
--health-interval 10s
73+
--health-timeout 5s
74+
--health-retries 5
75+
ports:
76+
- 9200:9200
77+
78+
apache-tika:
79+
image: ghcr.io/${{ github.repository }}/apache-tika:latest
80+
ports:
81+
- 9998:9998
82+
83+
steps:
84+
- uses: actions/checkout@v4
85+
86+
- name: Start MinIO
87+
run: |
88+
docker run -d \
89+
--name minio \
90+
--network host \
91+
--health-cmd "curl -f http://localhost:9000/minio/health/live || exit 1" \
92+
--health-interval 10s \
93+
--health-timeout 5s \
94+
--health-retries 5 \
95+
-e MINIO_ROOT_USER=minio-access-key \
96+
-e MINIO_ROOT_PASSWORD=minio-secret-key \
97+
quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z \
98+
server /data --console-address :9001
99+
100+
# Wait for MinIO to be ready
101+
echo "Waiting for MinIO to be ready..."
102+
timeout 60 bash -c 'until curl -sf http://localhost:9000/minio/health/live > /dev/null 2>&1; do sleep 2; done'
103+
echo "MinIO is ready"
104+
105+
- uses: actions/setup-python@v5
106+
with:
107+
python-version: '3.11'
108+
cache: 'pip'
109+
cache-dependency-path: requirements.txt
110+
- name: Install dependencies
111+
run: |
112+
python -m pip install --upgrade pip
113+
pip install -r requirements.txt
114+
115+
- name: Create MinIO bucket
116+
run: |
117+
# Wait for MinIO to be ready
118+
for i in {1..30}; do
119+
if curl -sf http://localhost:9000/minio/health/live > /dev/null 2>&1; then
120+
echo "MinIO is ready"
121+
break
122+
fi
123+
echo "Waiting for MinIO... ($i/30)"
124+
sleep 2
125+
done
126+
127+
# Detect architecture and download appropriate MinIO client
128+
ARCH=$(uname -m)
129+
if [ "$ARCH" = "aarch64" ] || [ "$ARCH" = "arm64" ]; then
130+
MC_URL="https://dl.min.io/client/mc/release/linux-arm64/mc"
131+
else
132+
MC_URL="https://dl.min.io/client/mc/release/linux-amd64/mc"
133+
fi
134+
135+
echo "Downloading MinIO client for architecture: $ARCH"
136+
curl -sLo /tmp/mc "$MC_URL"
137+
chmod +x /tmp/mc
138+
139+
# Configure MinIO client
140+
/tmp/mc alias set minio http://localhost:9000 minio-access-key minio-secret-key
141+
142+
# Create bucket if it doesn't exist
143+
/tmp/mc mb minio/queridodiariobucket --ignore-existing || true
144+
145+
echo "MinIO bucket created successfully"
146+
147+
- name: Create OpenSearch index
148+
timeout-minutes: 2
149+
run: |
150+
echo "=== Creating OpenSearch index ==="
151+
echo "OpenSearch host: http://localhost:9200"
152+
echo "Testing connectivity..."
153+
curl -v http://localhost:9200/_cluster/health || echo "OpenSearch not responding"
154+
echo "Running init script..."
155+
chmod +x init-scripts/opensearch/create-opensearch-index.sh
156+
OPENSEARCH_HOST=http://localhost:9200 \
157+
OPENSEARCH_USER=admin \
158+
OPENSEARCH_PASSWORD=admin \
159+
INDEX_NAME=querido-diario \
160+
./init-scripts/opensearch/create-opensearch-index.sh
161+
echo "=== OpenSearch index creation completed ==="
162+
163+
- name: Run unit tests
164+
timeout-minutes: 5
165+
env:
166+
PYTHONPATH: ${{ github.workspace }}
167+
PYTHONUNBUFFERED: 1
168+
POSTGRES_PASSWORD: queridodiario
169+
POSTGRES_USER: queridodiario
170+
POSTGRES_DB: queridodiariodb
171+
POSTGRES_HOST: localhost
172+
POSTGRES_PORT: 5432
173+
STORAGE_REGION: us-east-1
174+
STORAGE_ENDPOINT: http://localhost:9000
175+
STORAGE_ACCESS_KEY: minio-access-key
176+
STORAGE_ACCESS_SECRET: minio-secret-key
177+
STORAGE_BUCKET: queridodiariobucket
178+
OPENSEARCH_HOST: http://localhost:9200
179+
OPENSEARCH_INDEX: querido-diario
180+
OPENSEARCH_USER: admin
181+
OPENSEARCH_PASSWORD: admin
182+
APACHE_TIKA_SERVER: http://localhost:9998
183+
DEBUG: 1
184+
HF_HUB_OFFLINE: 1
185+
TRANSFORMERS_OFFLINE: 1
186+
SENTENCE_TRANSFORMERS_HOME: /tmp/sentence_transformers
187+
run: |
188+
echo "========================================="
189+
echo "🧪 STARTING INSTRUMENTED TEST RUN"
190+
echo "========================================="
191+
echo "Working directory: $(pwd)"
192+
echo "Python: $(which python)"
193+
echo "Python version: $(python --version)"
194+
echo "Time: $(date)"
195+
echo ""
196+
197+
# Make runner executable
198+
chmod +x instrumented_test_runner.py
199+
200+
# Run with our instrumented runner
201+
python instrumented_test_runner.py
202+
203+
echo ""
204+
echo "========================================="
205+
echo "Test run completed at: $(date)"
206+
echo "========================================="

.github/workflows_disabled/security_scan.yaml renamed to .github/workflows/security_scan.yaml_disabled

File renamed without changes.

0 commit comments

Comments
 (0)