Repo reorg to "postmodern" mono-layout

[diegoripley]

Overview

Restructure dataforcanada/d4c-infra-distribution into a maintainable "postmodern" mono-repo layout, establishing a simple, explicit, fast dev loop (uv + ruff + pytest + mypy) with clear boundaries between "apps" and "packages."

Split out from #26.

Goals

• Adopt a simple, explicit, fast dev loop with clear boundaries between "apps" and "packages."
• Turn experimental scripts into repeatable CLIs with tests and CI.
• Introduce an extensible directory structure for multiple ingestor protocols (HTTP(S), SFTP, BitTorrent, IPFS…).

Proposed repository layout

d4c-infra-distribution/
├─ apps/                      # runnable things (CLIs/services)
│  └─ ingestors/
│     ├─ http_cf/             # Cloudflare HTTP worker + Python caller
│     ├─ sftp/
│     ├─ ipfs/
│     └─ bittorrent/
├─ packages/                  # importable libs shared by apps
│  ├─ ingestion-core/         # retry/backoff, hashing, sqlite utils, parquet io
│  ├─ cf-common/              # Worker-side helpers: request validation, logging, R2 helpers
│  └─ ...
├─ infra/
│  ├─ cloudflare/             # wrangler.toml, R2 bucket config, queues, durable objects
│  ├─ terraform/              # IaC for CF + cloud infra (optional)
│  └─ k8s/                   # if/when needed
├─ data/                      # example small fixtures; no large data
├─ docs/                      # architecture notes, runbooks
├─ .github/
│  └─ workflows/              # CI: lint, type-check, test, build, deploy worker
├─ tools/                     # dev tooling (noxfile.py, codegen, lint hooks)
├─ pyproject.toml             # uv/ruff/pytest/mypy config in one place
├─ uv.lock
└─ README.md

Tasks

Repo reorg

• Create apps/, packages/, infra/, archive/, tools/, .github/workflows/
• Add root pyproject.toml with uv/ruff/mypy/pytest config; commit uv.lock
• Add noxfile.py with lint, typecheck, test, build sessions
• Move current content into the new layout with minimal path changes
• Document the new layout in README.md
• Write an ADR: "Adopt postmodern mono layout and ingestion architecture."
• Add runbooks for local dev, staging, prod, and recovery from partial failures

Acceptance criteria

• scripts/ contains only wrappers and a README mapping old → new paths
• CI runs ruff, mypy, pytest, and packages artifacts on PRs
• New layout is documented in README.md and an ADR under docs/

References

• https://github.com/carderne/postmodern-mono
• https://rdrn.me/postmodern-python/
• https://github.com/carderne/postmodern-python