Skip to content

Commit f29c0e9

Browse files
rluederrlueder
committed
ci: wire AWS OIDC + cron refresh completo + setup guide
1 parent 5f9248c commit f29c0e9

2 files changed

Lines changed: 296 additions & 8 deletions

File tree

‎.github/workflows/refresh.yml‎

Lines changed: 112 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,39 @@
11
name: refresh
22

3-
# Workflow de refresh manual: sonda FTP, arquiva delta em Parquet,
4-
# publica release. Só roda via workflow_dispatch — o cron automático
5-
# será reativado quando os secrets AWS (AWS_ROLE_ARN, S3_BUCKET,
6-
# CLOUDFRONT_DISTRIBUTION_ID) estiverem configurados no repo.
3+
# Sonda FTP DATASUS, arquiva delta em Parquet, emite provenance,
4+
# sincroniza com S3, invalida CloudFront, publica GitHub Release
5+
# (que dispara o webhook Zenodo → DOI).
6+
#
7+
# Cadência: segunda-feira 06:00 UTC. DATASUS publica SIA-PA com 45-90
8+
# dias de lag após fim da competência; sonda semanal cobre bem.
9+
#
10+
# Secrets necessários (configurados uma vez no repo):
11+
# AWS_ROLE_ARN — IAM role com acesso ao bucket + distribution
12+
# S3_BUCKET — nome do bucket S3 (sa-east-1)
13+
# CLOUDFRONT_DISTRIBUTION_ID — opcional, para invalidation
714

815
on:
16+
schedule:
17+
- cron: '0 6 * * 1'
918
workflow_dispatch:
19+
inputs:
20+
dryRun:
21+
description: 'Dry run — detect only, skip archive/upload'
22+
type: boolean
23+
default: false
1024

1125
permissions:
1226
contents: write
1327
id-token: write
1428

1529
jobs:
16-
refresh:
30+
detect:
1731
runs-on: ubuntu-latest
18-
timeout-minutes: 60
32+
timeout-minutes: 10
33+
outputs:
34+
hasNew: ${{ steps.detect.outputs.hasNew }}
35+
pendingCount: ${{ steps.detect.outputs.pendingCount }}
36+
latestCompetencia: ${{ steps.detect.outputs.latestCompetencia }}
1937
steps:
2038
- uses: actions/checkout@v4
2139
- uses: pnpm/action-setup@v4
@@ -24,5 +42,91 @@ jobs:
2442
node-version: 22
2543
cache: pnpm
2644
- run: pnpm install --frozen-lockfile
27-
- run: pnpm detect-new
28-
- run: echo 'TODO — wire archive / upload / release steps after secrets are set up'
45+
- id: detect
46+
run: pnpm detect-new
47+
- uses: actions/upload-artifact@v4
48+
with:
49+
name: pending
50+
path: state/pending.json
51+
retention-days: 7
52+
53+
archive:
54+
needs: detect
55+
if: needs.detect.outputs.hasNew == 'true' && inputs.dryRun != true
56+
runs-on: ubuntu-latest
57+
timeout-minutes: 180
58+
steps:
59+
- uses: actions/checkout@v4
60+
- uses: pnpm/action-setup@v4
61+
- uses: actions/setup-node@v4
62+
with:
63+
node-version: 22
64+
cache: pnpm
65+
- run: pnpm install --frozen-lockfile
66+
67+
- uses: actions/download-artifact@v4
68+
with:
69+
name: pending
70+
path: state/
71+
72+
- name: Archive SIA-PA delta (DBC → Parquet raw)
73+
run: pnpm archive-sia-pa
74+
75+
- name: Emit provenance (SHA256 + schema + metadata)
76+
run: pnpm emit-provenance
77+
78+
- name: Build manifest
79+
run: pnpm build-manifest
80+
81+
- uses: aws-actions/configure-aws-credentials@v4
82+
with:
83+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
84+
aws-region: sa-east-1
85+
86+
- name: Upload para S3
87+
run: |
88+
aws s3 sync build/ "s3://${{ secrets.S3_BUCKET }}/" \
89+
--exclude '_archive-run.json' \
90+
--exclude '**/*.ndjson' \
91+
--cache-control 'public, max-age=3600'
92+
93+
- name: Invalidate CloudFront
94+
if: ${{ secrets.CLOUDFRONT_DISTRIBUTION_ID != '' }}
95+
run: |
96+
aws cloudfront create-invalidation \
97+
--distribution-id "${{ secrets.CLOUDFRONT_DISTRIBUTION_ID }}" \
98+
--paths '/manifest.json' '/*/provenance/*'
99+
100+
- name: Atualizar state (merge pending → dataset state)
101+
run: pnpm detect-new -- --mark-processed
102+
103+
- name: Commit state atualizado
104+
run: |
105+
git config user.name 'datasus-parquet-bot'
106+
git config user.email 'noreply@precisa-saude.com.br'
107+
git add state/
108+
if git diff --cached --quiet; then
109+
echo 'state inalterado — skip commit'
110+
exit 0
111+
fi
112+
git commit -m "chore(state): refresh ${{ needs.detect.outputs.latestCompetencia }}"
113+
git push
114+
115+
- name: Criar GitHub Release (dispara webhook Zenodo → DOI)
116+
uses: softprops/action-gh-release@v2
117+
with:
118+
tag_name: dataset-${{ needs.detect.outputs.latestCompetencia }}
119+
name: Dataset ${{ needs.detect.outputs.latestCompetencia }}
120+
body: |
121+
Refresh automático: **${{ needs.detect.outputs.pendingCount }}** novas competências
122+
(mais recente: **${{ needs.detect.outputs.latestCompetencia }}**).
123+
124+
Assets incluem `manifest.json`, Parquet por mês e provenance por partição.
125+
Para validar byte-a-byte contra o FTP oficial, veja `docs/provenance.md`.
126+
127+
DOI Zenodo emitido automaticamente via webhook — veja
128+
https://zenodo.org/search?q=Precisa-Saude/datasus-parquet.
129+
files: |
130+
build/manifest.json
131+
build/**/provenance/**/*.json
132+
generate_release_notes: true

‎docs/aws-setup.md‎

Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# Setup AWS — bucket, CloudFront, OIDC role
2+
3+
Configuração única de infraestrutura pra permitir que o workflow
4+
`refresh.yml` publique raw Parquet em produção. Depois de feito,
5+
roda automaticamente via cron semanal.
6+
7+
## 1. Criar bucket S3
8+
9+
Região **sa-east-1** (latência pra Brasil + consistência com
10+
`datasus-viz`).
11+
12+
```bash
13+
aws s3api create-bucket \
14+
--bucket datasus-parquet \
15+
--region sa-east-1 \
16+
--create-bucket-configuration LocationConstraint=sa-east-1
17+
```
18+
19+
**CORS permissivo** (leitura pública, DuckDB WASM faz Range requests):
20+
21+
```bash
22+
aws s3api put-bucket-cors --bucket datasus-parquet --cors-configuration '{
23+
"CORSRules": [{
24+
"AllowedOrigins": ["*"],
25+
"AllowedMethods": ["GET", "HEAD"],
26+
"AllowedHeaders": ["Range", "If-None-Match", "If-Match"],
27+
"ExposeHeaders": ["Content-Range", "Content-Length", "ETag"],
28+
"MaxAgeSeconds": 3000
29+
}]
30+
}'
31+
```
32+
33+
**Bucket policy** (leitura anônima, sem PUT público):
34+
35+
```bash
36+
aws s3api put-bucket-policy --bucket datasus-parquet --policy '{
37+
"Version": "2012-10-17",
38+
"Statement": [{
39+
"Sid": "PublicRead",
40+
"Effect": "Allow",
41+
"Principal": "*",
42+
"Action": ["s3:GetObject"],
43+
"Resource": "arn:aws:s3:::datasus-parquet/*"
44+
}]
45+
}'
46+
```
47+
48+
## 2. CloudFront distribution
49+
50+
Cria CDN em frente ao bucket pra absorver cache hits e reduzir custo
51+
de Range requests repetidos.
52+
53+
```bash
54+
aws cloudfront create-distribution --distribution-config '{
55+
"CallerReference": "datasus-parquet-2026-04",
56+
"Comment": "datasus-parquet public archive",
57+
"Enabled": true,
58+
"Origins": {"Quantity":1,"Items":[{
59+
"Id": "s3-datasus-parquet",
60+
"DomainName": "datasus-parquet.s3.sa-east-1.amazonaws.com",
61+
"S3OriginConfig": {"OriginAccessIdentity": ""}
62+
}]},
63+
"DefaultCacheBehavior": {
64+
"TargetOriginId": "s3-datasus-parquet",
65+
"ViewerProtocolPolicy": "redirect-to-https",
66+
"AllowedMethods": {"Quantity": 2, "Items": ["GET","HEAD"]},
67+
"CachePolicyId": "658327ea-f89d-4fab-a63d-7e88639e58f6"
68+
},
69+
"PriceClass": "PriceClass_100"
70+
}'
71+
```
72+
73+
Anotar o `Id` e `DomainName` retornados — `DomainName` é o URL público
74+
(formato `dXXXXXXXXX.cloudfront.net`) usado pelos consumidores.
75+
76+
## 3. IAM role pra OIDC (GitHub Actions → AWS sem credenciais fixas)
77+
78+
### 3.1. Provider OIDC do GitHub (se ainda não existir na conta)
79+
80+
```bash
81+
aws iam create-open-id-connect-provider \
82+
--url https://token.actions.githubusercontent.com \
83+
--client-id-list sts.amazonaws.com \
84+
--thumbprint-list 6938fd4d98bab03faadb97b34396831e3780aea1
85+
```
86+
87+
### 3.2. Role de publish
88+
89+
Trust policy restringe ao repo específico:
90+
91+
```bash
92+
cat > trust-policy.json <<'EOF'
93+
{
94+
"Version": "2012-10-17",
95+
"Statement": [{
96+
"Effect": "Allow",
97+
"Principal": {
98+
"Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
99+
},
100+
"Action": "sts:AssumeRoleWithWebIdentity",
101+
"Condition": {
102+
"StringEquals": {
103+
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
104+
},
105+
"StringLike": {
106+
"token.actions.githubusercontent.com:sub":
107+
"repo:Precisa-Saude/datasus-parquet:*"
108+
}
109+
}
110+
}]
111+
}
112+
EOF
113+
aws iam create-role \
114+
--role-name datasus-parquet-publish \
115+
--assume-role-policy-document file://trust-policy.json
116+
```
117+
118+
### 3.3. Permissions policy (least-privilege)
119+
120+
```bash
121+
cat > publish-policy.json <<'EOF'
122+
{
123+
"Version": "2012-10-17",
124+
"Statement": [
125+
{
126+
"Sid": "S3Write",
127+
"Effect": "Allow",
128+
"Action": ["s3:PutObject", "s3:DeleteObject", "s3:ListBucket"],
129+
"Resource": [
130+
"arn:aws:s3:::datasus-parquet",
131+
"arn:aws:s3:::datasus-parquet/*"
132+
]
133+
},
134+
{
135+
"Sid": "CloudFrontInvalidate",
136+
"Effect": "Allow",
137+
"Action": "cloudfront:CreateInvalidation",
138+
"Resource": "arn:aws:cloudfront::ACCOUNT_ID:distribution/DISTRIBUTION_ID"
139+
}
140+
]
141+
}
142+
EOF
143+
aws iam put-role-policy \
144+
--role-name datasus-parquet-publish \
145+
--policy-name datasus-parquet-publish-policy \
146+
--policy-document file://publish-policy.json
147+
```
148+
149+
Anotar o `arn:aws:iam::ACCOUNT_ID:role/datasus-parquet-publish`.
150+
151+
## 4. Secrets no repo GitHub
152+
153+
```bash
154+
gh secret set AWS_ROLE_ARN \
155+
--repo Precisa-Saude/datasus-parquet \
156+
--body "arn:aws:iam::ACCOUNT_ID:role/datasus-parquet-publish"
157+
158+
gh secret set S3_BUCKET \
159+
--repo Precisa-Saude/datasus-parquet \
160+
--body "datasus-parquet"
161+
162+
gh secret set CLOUDFRONT_DISTRIBUTION_ID \
163+
--repo Precisa-Saude/datasus-parquet \
164+
--body "DISTRIBUTION_ID_AQUI"
165+
```
166+
167+
## 5. Smoke test
168+
169+
Disparar o workflow manual via `workflow_dispatch` com `dryRun=true`:
170+
171+
```bash
172+
gh workflow run refresh.yml \
173+
--repo Precisa-Saude/datasus-parquet \
174+
-f dryRun=true
175+
```
176+
177+
Isso roda só o step `detect` — se passar, a configuração tá OK. Depois
178+
rodar sem `dryRun` pra primeira ingestão real.
179+
180+
## 6. Monitoramento
181+
182+
- `gh run list --repo Precisa-Saude/datasus-parquet --workflow refresh.yml`
183+
- CloudWatch métricas do bucket (puts, gets, bandwidth)
184+
- CloudFront cache hit ratio

0 commit comments

Comments
 (0)