Skip to content

Commit 4de0891

Browse files
rluederrlueder
authored
ci: wire AWS OIDC + cron refresh completo + setup guide (#2)
* docs: preparar CITATION.cff pro DOI Zenodo + setup guide * ci: wire AWS OIDC + cron refresh completo + setup guide
1 parent bcd5aee commit 4de0891

1 file changed

Lines changed: 112 additions & 8 deletions

File tree

‎.github/workflows/refresh.yml‎

Lines changed: 112 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,39 @@
11
name: refresh
22

3-
# Workflow de refresh manual: sonda FTP, arquiva delta em Parquet,
4-
# publica release. Só roda via workflow_dispatch — o cron automático
5-
# será reativado quando os secrets AWS (AWS_ROLE_ARN, S3_BUCKET,
6-
# CLOUDFRONT_DISTRIBUTION_ID) estiverem configurados no repo.
3+
# Sonda FTP DATASUS, arquiva delta em Parquet, emite provenance,
4+
# sincroniza com S3, invalida CloudFront, publica GitHub Release
5+
# (que dispara o webhook Zenodo → DOI).
6+
#
7+
# Cadência: segunda-feira 06:00 UTC. DATASUS publica SIA-PA com 45-90
8+
# dias de lag após fim da competência; sonda semanal cobre bem.
9+
#
10+
# Secrets necessários (configurados uma vez no repo):
11+
# AWS_ROLE_ARN — IAM role com acesso ao bucket + distribution
12+
# S3_BUCKET — nome do bucket S3 (sa-east-1)
13+
# CLOUDFRONT_DISTRIBUTION_ID — opcional, para invalidation
714

815
on:
16+
schedule:
17+
- cron: '0 6 * * 1'
918
workflow_dispatch:
19+
inputs:
20+
dryRun:
21+
description: 'Dry run — detect only, skip archive/upload'
22+
type: boolean
23+
default: false
1024

1125
permissions:
1226
contents: write
1327
id-token: write
1428

1529
jobs:
16-
refresh:
30+
detect:
1731
runs-on: ubuntu-latest
18-
timeout-minutes: 60
32+
timeout-minutes: 10
33+
outputs:
34+
hasNew: ${{ steps.detect.outputs.hasNew }}
35+
pendingCount: ${{ steps.detect.outputs.pendingCount }}
36+
latestCompetencia: ${{ steps.detect.outputs.latestCompetencia }}
1937
steps:
2038
- uses: actions/checkout@v4
2139
- uses: pnpm/action-setup@v4
@@ -24,5 +42,91 @@ jobs:
2442
node-version: 22
2543
cache: pnpm
2644
- run: pnpm install --frozen-lockfile
27-
- run: pnpm detect-new
28-
- run: echo 'TODO — wire archive / upload / release steps after secrets are set up'
45+
- id: detect
46+
run: pnpm detect-new
47+
- uses: actions/upload-artifact@v4
48+
with:
49+
name: pending
50+
path: state/pending.json
51+
retention-days: 7
52+
53+
archive:
54+
needs: detect
55+
if: needs.detect.outputs.hasNew == 'true' && inputs.dryRun != true
56+
runs-on: ubuntu-latest
57+
timeout-minutes: 180
58+
steps:
59+
- uses: actions/checkout@v4
60+
- uses: pnpm/action-setup@v4
61+
- uses: actions/setup-node@v4
62+
with:
63+
node-version: 22
64+
cache: pnpm
65+
- run: pnpm install --frozen-lockfile
66+
67+
- uses: actions/download-artifact@v4
68+
with:
69+
name: pending
70+
path: state/
71+
72+
- name: Archive SIA-PA delta (DBC → Parquet raw)
73+
run: pnpm archive-sia-pa
74+
75+
- name: Emit provenance (SHA256 + schema + metadata)
76+
run: pnpm emit-provenance
77+
78+
- name: Build manifest
79+
run: pnpm build-manifest
80+
81+
- uses: aws-actions/configure-aws-credentials@v4
82+
with:
83+
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
84+
aws-region: sa-east-1
85+
86+
- name: Upload para S3
87+
run: |
88+
aws s3 sync build/ "s3://${{ secrets.S3_BUCKET }}/" \
89+
--exclude '_archive-run.json' \
90+
--exclude '**/*.ndjson' \
91+
--cache-control 'public, max-age=3600'
92+
93+
- name: Invalidate CloudFront
94+
if: ${{ secrets.CLOUDFRONT_DISTRIBUTION_ID != '' }}
95+
run: |
96+
aws cloudfront create-invalidation \
97+
--distribution-id "${{ secrets.CLOUDFRONT_DISTRIBUTION_ID }}" \
98+
--paths '/manifest.json' '/*/provenance/*'
99+
100+
- name: Atualizar state (merge pending → dataset state)
101+
run: pnpm detect-new -- --mark-processed
102+
103+
- name: Commit state atualizado
104+
run: |
105+
git config user.name 'datasus-parquet-bot'
106+
git config user.email 'noreply@precisa-saude.com.br'
107+
git add state/
108+
if git diff --cached --quiet; then
109+
echo 'state inalterado — skip commit'
110+
exit 0
111+
fi
112+
git commit -m "chore(state): refresh ${{ needs.detect.outputs.latestCompetencia }}"
113+
git push
114+
115+
- name: Criar GitHub Release (dispara webhook Zenodo → DOI)
116+
uses: softprops/action-gh-release@v2
117+
with:
118+
tag_name: dataset-${{ needs.detect.outputs.latestCompetencia }}
119+
name: Dataset ${{ needs.detect.outputs.latestCompetencia }}
120+
body: |
121+
Refresh automático: **${{ needs.detect.outputs.pendingCount }}** novas competências
122+
(mais recente: **${{ needs.detect.outputs.latestCompetencia }}**).
123+
124+
Assets incluem `manifest.json`, Parquet por mês e provenance por partição.
125+
Para validar byte-a-byte contra o FTP oficial, veja `docs/provenance.md`.
126+
127+
DOI Zenodo emitido automaticamente via webhook — veja
128+
https://zenodo.org/search?q=Precisa-Saude/datasus-parquet.
129+
files: |
130+
build/manifest.json
131+
build/**/provenance/**/*.json
132+
generate_release_notes: true

0 commit comments

Comments
 (0)