scdd-rewrite — a reproducible port of the supply-chain due-diligence pipeline

A self-contained Python + Julia port of src/scrc_freeze/, the frozen pipeline behind the published supply-chain due-diligence results. Inputs are pinned, every stage is numbered in DAG order, and one run id (r01..r10) maps to one process.

What this pipeline does

M0  oecd_prep              derive df_io / df_countries / df_ind / df_nind from raw OECD ICIO 2021
M1  eurostat_fill          impute missing turnover / headcount / firm counts (one-shot)
M2  comp_sampling          synthetic firm sample per iteration (seeded)
M3  io_alignment           align comp samples with the OECD I/O matrix
M4  network_reconstruct    stochastic firm-level edge sampling (Julia, threaded)
M5  csv2parquet            convert M4 outputs from CSV to Parquet
M6  eu_enrichment          attach allegations to EU buyers
M7  impacts                5-tier sparse-matrix propagation; per-iter pickle
M8  netw_stats             degree / link / diameter statistics
M9  urspruenge             origin tracing (where do allegations come from)
M10 plots_final            aggregate all rNN into the published risk-profile PDFs
M11 non_eu_instantiate     synthetic non-EU supplier firms (per iteration)
M12 non_eu_coverage        node-vs-link SCDD coverage curves
M13 non_eu_plots           certificates-vs-SCDD comparison PDFs

10 iterations r01..r10 are run sequentially (one process per iteration — this is how the memory ceiling is enforced). M0, M1, M10, M13 are one-shot stages whose output is shared across iterations.

Layout

.
├── README.md                 — this file
├── Makefile                  — convenience targets (run, test, list-stages)
├── pyproject.toml
├── inputs/                   — external data the pipeline reads (see inputs/README.md)
├── julia/                    — Julia package for M4
├── scripts/                  — helper shell scripts (memory watch, resource log)
├── src/scdd_rewrite/         — Python source
│   ├── cli.py                — `scdd-rewrite` CLI
│   ├── orchestrate.py        — stage registry + per-iteration loop
│   ├── config.py             — thresholds, EU set, defaults
│   ├── manifest.py           — per-stage manifest writer (input/output hashes, seed)
│   ├── paths.py              — output directory policy
│   ├── schemas.py            — column / dtype contracts
│   ├── tables.py             — small CSV/parquet helpers
│   └── stages/               — one module per milestone, prefixed mNN_ so `ls` shows DAG order
└── tests/unit/               — unit tests for config, paths, manifest, schemas, tables

Inputs

All external data the pipeline reads lives under inputs/. 17 files, ~180 MB:

Subdir	What	Consumer
inputs/eurostat/	7 raw Eurostat tables (turnover, employees, size classes)	M1
inputs/oecd/	OECD ICIO 2021 release for reference year 2018 (raw + derivative)	M0 (raw), M3 (derivative as fallback)
inputs/helper_files/	hand-crafted lookups (risk-sector flag, 10-char industry labels)	M3, M6, M7, M8, M9, M10
inputs/imports_allegations/	allegation counts + EU27 trade fractions	M6

See inputs/README.md for the full file-by-file manifest and provenance.

Running

make setup           # install scdd-rewrite into the active Python env (editable)
make test            # run unit tests
make list-stages     # print M0..M13 names with descriptions
make run RUN=r01     # one full per-iteration pipeline run
make run-all         # r01..r10 sequentially

Direct CLI:

python -m scdd_rewrite.cli list-stages
python -m scdd_rewrite.cli run --run-id r01 \
    --input-root inputs --output-root runs/full/r01
python -m scdd_rewrite.cli run-stage --run-id r01 --stage oecd_prep \
    --input-root inputs --output-root runs/full/r01

Outputs land in runs/<scope>/<run_id>/ plus the two shared dirs runs/<scope>/oecd_prep/ (M0) and runs/<scope>/eurostat/ (M1). One-shot stages M10 and M13 write to runs/<scope>/aggregate/.

The non-negotiable memory constraint

The freeze pipeline is shaped by memory and throughput limits, not abstract ergonomics. Reconstructed networks are ~10⁹ edges per iteration; one prepared-data CSV is ~16 GB; a full reconstruction directory is 269 GB. The per-iteration argv pattern — one rNN per process — is how the wall-clock memory ceiling is enforced. Two concurrent iterations on one host will OOM.

A naive port that "modernises" the points below without measuring will OOM. Each item is a load-bearing invariant of the freeze pipeline that the rewrite preserves on purpose.

#	Invariant	Where
R1	Per-iteration argv pattern: one rNN per process, orchestrator loops externally	orchestrate.py, cli.py
R2	Bounded-batch link adder with explicit cap = 5×10⁷ links per iteration	M4 (Julia)
R3	Floor + Bernoulli rounding in get_A_slice (unbiased discretisation of n_links · A)	M4 (Julia)
R4	Asymmetric EU vs non-EU outgoing-link assignment	M4 (Julia)
R5	Per-group dict layout for netw (not a sparse matrix)	M4 (Julia)
R6	netw::Dict{Tuple{Int,Int},Bool} drained edge-by-edge with delete! to keep RSS flat during write	M4 (Julia)
R7	Phase-separated writes with explicit nullification of intermediate arrays	M4 (Julia)
R8	Aggressive GC.gc() inside the threaded loop	M4 (Julia)
R9	Threaded loop with per-iteration shuffle (load balance)	M4 (Julia)
R10	CSV+gzip output from Julia, not parquet (M5 converts later)	M4 → M5
R11	Polars eager + explicit del + gc.collect() after every large join	M3, M6, M7, M9 (Python)
R12	Sparse boolean tier propagation via setdiag(1) + .T.dot(xi) — never A[i,i] = 1.0	M7 (Python)
R13	CSV → parquet conversion as a barrier stage between Julia and Python	M5 (Python)
R14	Pickle dict-of-dataframes for impact aggregations (one process writes, M10 reads)	M7, M10 (Python)

Ground rules

• No script writes outside runs/. Enforced by paths.py.
• Every run drops a manifest with seed, input hashes, output hashes, elapsed time.
• Stochastic stages are seeded (config.default_seed_for_run).
• No modernisation for its own sake. Every change to a memory-sensitive primitive needs a measured peak-RSS comparison before landing.