This repository contains the reproducible pipeline for analysing gender representation in UK REF 2021 Impact Case Studies (ICS) and REF-linked research outputs.
The pipeline builds:
- case-level ICS gender counts from extracted staff names,
- output-level author gender counts from Dimensions metadata,
- regex and OpenAI thematic indicators from REF case-study text,
- model-comparison diagnostics across thematic classifiers,
- regression models, manuscript figures, tables, and supplementary text-analysis outputs.
All commands below are intended to be run from the repository root.
Core pipeline scripts:
src/step01_make_enhanced_data.pybuildsdata/analysis/enhanced_ref_data.{parquet,csv}and manages OpenAI thematic-indicator cache rows indata/openai/categories.csv.src/step02_make_ref_staff.pydownloads/caches ICS PDFs, extracts staff blocks, uses LLM extraction where enabled, and writesdata/ics_staff_rows/.src/step03_get_dimensions_research_outputs.pyqueries or reuses Dimensions chunks and writesdata/analysis/outputs_concat_with_*_authors.{parquet,csv}.src/step04_make_figure_one.pyrebuilds Figure 1.src/step05_build_regression_models.pyfits regression models and writesoutputs/models/regression_results.pkl.src/step06_make_figure_two.pyrebuilds Figure 2 plus supplementary regression figures.src/step07_make_table_one.pywritesoutputs/tables/regression_results.tex.src/step08_build_statistics.pywrites descriptive statistics and summary tables.src/step09_evaluate_thematic_indicators.pycompares regex, GPT-5-nano, GPT-5.1, GPT-5.4, and GPT-5.5 thematic indicators.src/step10_analyze_ics_text_gender.pyruns word-level ICS text/gender association analysis.
Important directories:
data/source/: downloaded source workbooks and cached Dimensions chunks.data/cache/ref_pdfs/: cached ICS PDFs.data/ics_staff_rows/: staff extraction rows, case-level counts, and audit files.data/openai/: OpenAI thematic cache.data/analysis/: canonical analysis-ready datasets.data/final/: final enhanced dataset exports.data/dimensions_outputs/: legacy mirror of Dimensions output-author tables.outputs/figures/: figures in PDF/SVG/PNG.outputs/tables/: manuscript and diagnostic tables.outputs/models/: fitted model artifacts.
Configuration is in pipeline.yaml.
Current OpenAI defaults:
- primary model:
gpt-5.5 - prompt version:
v2 - service tier:
flex - processing mode:
sync - OpenAI Batch API: not used by default
- thematic request batch size:
1 - staff extraction request batch size:
1 - transient OpenAI retry budget for staff extraction:
5 - prompt cache key:
thematic_indicators_v2
Thematic classification uses all five REF ICS text fields:
1. Summary of the impact2. Underpinning research3. References to the research4. Details of the impact5. Sources to corroborate the impact
Staff extraction is robust by design:
- PDF text extraction is cached under
data/cache/ref_pdfs/. - LLM extraction retries transient OpenAI failures.
- Cases that remain genuinely unresolved are kept as unresolved/blank counts, not silently converted to zero people.
step02.require_people: falseallows the pipeline to continue while preserving unresolved status in audit outputs.
Gender inference for both ICS staff names and Dimensions author forenames uses:
gender_guesseras the primary deterministic classifier.gender_detectoras a fallback only whengender_guesserreturnsunknown.
The stored output labels remain exactly:
malefemaleunknown
OpenAI credentials can be provided by either:
- environment variable:
OPENAI_API_KEY - file fallback:
keys/OPENAI_API_KEY
Dimensions credentials can be provided by either:
- environment variable:
DIMENSIONS_API_KEY - file fallback:
keys/dimensions_apikey.txt
keys/ is intentionally preserved by rebuild commands.
Install dependencies:
pip install -r requirements.txtIf parquet read/write fails, check that pyarrow is installed.
Use this when local data/cache artifacts already exist and you do not want new API calls:
./rerun_pipeline.sh --mode offlineThis reuses existing enhanced data, staff rows, OpenAI cache rows, and Dimensions chunks/outputs. It then rebuilds downstream figures, tables, models, and diagnostics.
Use this when data/analysis/ is already current and only outputs need rebuilding:
./rerun_pipeline.sh --mode downstreamThis runs steps 04-10 only.
Use this when you want to refresh source data and external-service outputs while preserving cached/generated directories:
./rerun_pipeline.sh --mode refreshThis may call REF downloads, OpenAI, and Dimensions.
Use only when you want to delete generated data and rebuild from scratch:
./rerun_pipeline.sh --mode rebuild --i-understand-this-deletes-dataThis deletes generated data/output directories, preserves data/manual/ and keys/, then runs the refresh pipeline.
Use when the primary enhanced dataset and staff rows already exist, but comparison-model thematic cache rows still need filling:
./rerun_pipeline.sh --mode resume-flexThis backfills:
gpt-5-nanogpt-5.1gpt-5.4
using flex-tier synchronous calls with one item per request, then continues through Dimensions and downstream steps.
If Dimensions API chunking completed but step03 failed while assembling or writing outputs, do not rerun expensive API calls. Rebuild from saved raw chunks:
python -m src.step03_get_dimensions_research_outputs \
--config pipeline.yaml \
--project-root . \
--skip-api \
--forceThen continue:
./rerun_pipeline.sh --mode downstreamOne-line version:
mkdir -p logs && { python -m src.step03_get_dimensions_research_outputs --config pipeline.yaml --project-root . --skip-api --force && ./rerun_pipeline.sh --mode downstream; } 2>&1 | tee logs/resume_from_step03_$(date +%Y%m%d_%H%M%S).logThe current step03 implementation serializes complex Dimensions fields such as authors and category_for_2020 as JSON strings before writing parquet. This avoids PyArrow failures caused by mixed nested types in Dimensions metadata.
Dimensions raw chunks:
find data/source/dimensions_api/raw -name 'df_*.csv' -printf '%h\n' | sort | uniq -cOpenAI thematic cache:
python - <<'PY'
import pandas as pd
df = pd.read_csv("data/openai/categories.csv", usecols=["model", "prompt_version", "llm_status", "cache_key"])
print(df.groupby(["model", "prompt_version", "llm_status"]).size())
print("bad rows:", len(df[~df["llm_status"].isin(["ok", "ok_prompt_cache_fallback"])]))
print(df.groupby(["model", "prompt_version"])["cache_key"].nunique())
PY| Step | Script | External calls by default | Primary outputs |
|---|---|---|---|
| 01 | step01_make_enhanced_data.py |
REF downloads; OpenAI when --with-llm needs uncached rows |
data/analysis/enhanced_ref_data.*, data/openai/categories.csv |
| 02 | step02_make_ref_staff.py |
ICS PDF downloads; OpenAI when --with-llm needs extraction |
data/ics_staff_rows/*.csv |
| 03 | step03_get_dimensions_research_outputs.py |
Dimensions unless --skip-api |
data/analysis/outputs_concat_*.{parquet,csv} |
| 04 | step04_make_figure_one.py |
none | outputs/figures/figure_one.{pdf,svg,png} |
| 05 | step05_build_regression_models.py |
none | outputs/models/regression_results.pkl |
| 06 | step06_make_figure_two.py |
none | outputs/figures/figure_two.*, supplementary regression figures |
| 07 | step07_make_table_one.py |
none | outputs/tables/regression_results.tex |
| 08 | step08_build_statistics.py |
none | outputs/tables/statistics_report.txt, summary .tex tables |
| 09 | step09_evaluate_thematic_indicators.py |
none | thematic diagnostics and comparison figures |
| 10 | step10_analyze_ics_text_gender.py |
none | supp_figure_5.*, word-association tables |
Step09 compares five method families:
- Regex rules.
- GPT-5-nano, displayed in some outputs as
GPT-5-mini. - GPT-5.1.
- GPT-5.4.
- GPT-5.5, the primary model used by the main enhanced dataset and Figure 2.
If comparison cache slices need filling manually:
python -m src.step01_make_enhanced_data --config pipeline.yaml --project-root . --backfill-model gpt-5-nano --backfill-prompt-version v2 --backfill-service-tier flex --backfill-batch-size 1 --backfill-prompt-cache-key thematic_indicators_v2
python -m src.step01_make_enhanced_data --config pipeline.yaml --project-root . --backfill-model gpt-5.1 --backfill-prompt-version v2 --backfill-service-tier flex --backfill-batch-size 1 --backfill-prompt-cache-key thematic_indicators_v2
python -m src.step01_make_enhanced_data --config pipeline.yaml --project-root . --backfill-model gpt-5.4 --backfill-prompt-version v2 --backfill-service-tier flex --backfill-batch-size 1 --backfill-prompt-cache-key thematic_indicators_v2Then rebuild the enhanced dataset if needed:
python -m src.step01_make_enhanced_data --config pipeline.yaml --project-root . --with-llm --forceFigures:
outputs/figures/figure_one.{pdf,svg,png}outputs/figures/figure_two.{pdf,svg,png}outputs/figures/supp_figure_2.{pdf,svg,png}outputs/figures/supp_figure_3.{pdf,svg,png}outputs/figures/supp_figure_4.{pdf,svg,png}outputs/figures/sup_figure_1.{pdf,svg,png}outputs/figures/supp_figure_5.{pdf,svg,png}
Tables:
outputs/tables/regression_results.texoutputs/tables/panel_summary.texoutputs/tables/uoa_summary.texoutputs/tables/llm_summary.texoutputs/tables/llm_panel_summary.texoutputs/tables/statistics_report.txtoutputs/tables/thematic_model_health_checks.csvoutputs/tables/thematic_pairwise_agreement_by_topic.csvoutputs/tables/thematic_pairwise_agreement_summary.csvoutputs/tables/thematic_topic_positive_rates.csvoutputs/tables/thematic_adjudication_sample.csvoutputs/tables/supp_figure_5_all.csvoutputs/tables/supp_figure_5_top_positive.csvoutputs/tables/supp_figure_5_top_negative.csv
Models:
outputs/models/regression_results.pkl
Run targeted tests for the recent pipeline fixes:
python -m unittest tests.test_step03_gender_inference tests.test_pipeline_schemaCheck the latest thematic method coverage:
cat outputs/tables/thematic_model_health_checks.csvCheck Dimensions output row contracts:
python - <<'PY'
import pandas as pd
any_df = pd.read_parquet("data/analysis/outputs_concat_with_any_number_authors.parquet")
pos_df = pd.read_parquet("data/analysis/outputs_concat_with_positive_authors.parquet")
print("any rows:", len(any_df))
print("positive rows:", len(pos_df))
print("positive ids subset of any ids:", set(pos_df["REF2ID"].astype(str)).issubset(set(any_df["REF2ID"].astype(str))))
print("zero-author rows in any:", int((pd.to_numeric(any_df["number_people"], errors="coerce").fillna(-1) == 0).sum()))
print("non-positive rows in positive:", int((pd.to_numeric(pos_df["number_people"], errors="coerce").fillna(0) <= 0).sum()))
PY- OpenAI thematic cache keys are deterministic hashes of model, prompt version, and normalized five-field ICS text.
- Successful prompt-cache fallback rows are stored with
llm_status=ok_prompt_cache_fallbackand are treated as valid completions. step03 --skip-apireuses saved Dimensions chunks and does not call Dimensions.- Step09 strict health checks fail on missing, disabled, parse-error, or otherwise bad thematic rows.
rerun_pipeline.shsetsREF_SKIP_MANIFEST=1by default, so manifest rows may be absent unless this is overridden.
GNU GPLv3. See LICENSE.