Selection Shapes the Boundary

Reproduction package for "Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations".

The study asks whether a monotonicity distinction that separated high-disagreement items in a selected corpus still separates them in the unselected populations those corpora were drawn from. It registers a directional prediction, then reports that the prediction fails. The non-upward group does not sit lower on the agreement ordinal. All seven contrasts come out with the opposite sign, and every one of them falls below the registered smallest effect size of interest. The two confirmatory contrasts are SNLI +0.0318 with a bootstrap interval of [-0.0105, +0.0749] and a Holm-corrected p of 0.162, and MNLI matched +0.0450 with [+0.0242, +0.0658] and a Holm-corrected p of 7.12e-05. The matched contrast reaches significance in the direction opposite to the registration, which we report as a failure of the directional hypothesis rather than as a finding.

Disagreement here is a measurement target, not noise to be removed. The monotonicity predictor is produced by operator-constrained label assignment, a rule-based tagger whose scope of application is fixed in advance, and the package includes the human validation of that tagger rather than treating its output as settled.

AUDIT.md carries the registration timeline, every deviation, and every negative result. Read it alongside this file.

What this repository does and does not contain

It contains the analysis code, the human Tier 3 labels, the figures, and a fetch script pinned to specific dataset versions.

It does not redistribute SNLI, MultiNLI, ChaosNLI, or MED. SNLI is CC BY-SA 4.0, whose share-alike term does not compose with this repository's MIT license, and MultiNLI carries no single license grant. Run scripts/fetch_data.sh to obtain the corpora from their original distributors.

It is a curated snapshot, not the full development history of the project. AUDIT.md says so in its own terms and explains what that costs.

Its registered scope is the plan document of 2026-07-17, which covers the two confirmatory contrasts, the secondary mismatched contrast, and the sensitivity analyses set out in AUDIT.md. Nothing else in this repository is registered. The scripts under analysis/exploratory/ sit outside that scope by construction and carry their own statement to that effect.

Setup

Python 3.13.9.

pip install -r requirements.txt
python -m spacy download en_core_web_trf   # 3.8.0

requirements.txt pins the exact versions the published numbers were computed with. The spaCy transformer model is a separate download and is required by analysis/operator_tagger.py.

Data

bash scripts/fetch_data.sh downloads all four datasets into data/raw/. Re-running skips anything already present.

Dataset	Source	Version	License
SNLI 1.0	nlp.stanford.edu/projects/snli/snli_1.0.zip	1.0	CC BY-SA 4.0
MultiNLI 1.0	cims.nyu.edu/~sbowman/multinli/multinli_1.0.zip	1.0	No single grant. Mixed OANC, CC-family, and public domain, as stated in the corpus paper
ChaosNLI	github.com/easonnie/ChaosNLI distribution link	v1.0	CC BY-NC 4.0, per the bundled README.txt
MED	github.com/verypluming/MED	commit 860ba36ce95ecba9f1ebe348d3d8cf1b3cba435a	CC BY-SA 4.0

ChaosNLI's CC BY-NC term is one more reason no corpus text is redistributed here.

The fetch script prints the sha256 of each archive and of six key files after download. It does not fail on a mismatch, so the comparison is yours to make. The values recorded when the published numbers were computed, first 16 hex characters:

File	sha256 prefix
snli.zip	afb3d70a5af5d8de
multinli.zip	049f507b9e36b1fc
chaosnli.zip	8ba7a6632aeb3a64
snli_1.0_dev.jsonl	9c03faff70182ef0
multinli_1.0_dev_matched.jsonl	9db14e02591c51b7
multinli_1.0_dev_mismatched.jsonl	65f3a993919f6a26
chaosNLI_snli.jsonl	99f9015ddda7d85f
chaosNLI_mnli_m.jsonl	8eb49b589488e7b1
MED.tsv	644d4ffc16fffabb

If a prefix does not match, the upstream distribution has changed since 2026-07-17 and the numbers below may not reproduce. Report the mismatch rather than working around it.

Tier 3 human labels

data/tier3_manual_labels.tsv is the one artifact here that no seed can regenerate. It holds 400 human monotonicity judgments, 200 from each of two independently drawn rounds, with columns pairID, cell, corpus, round, and manual_mono. There is deliberately no premise or hypothesis column, so that the labels can be published without redistributing corpus text.

Both rounds are kept. Round 1 agreed with the tagger at 27/200 = 0.1350, which is chance level, because the blind sheet failed to fix the judgment target to the hypothesis alone. That is recorded as a specification failure in the validation instrument, not as tagger error, and it is not dropped. After a codebook fixed the judgment target, round 2 reached 175/200 = 0.8750 with Cohen's kappa 0.6075. Round 2 is a different 200-item sample, not the same items tagged again; the two rounds share no pairID.

Step 16 of the run order rebuilds the filled tagging sheets locally by joining these labels onto the blind-sheet text you regenerate in steps 14 and 15. Steps 17 to 19 then run unchanged. This path was verified: both tier3_agreement.json and tier3_resheet_agreement.json rebuilt this way are identical to the committed results, ignoring only the generation timestamp.

Run order

All commands assume the repository root as the working directory. The dependency notes come from each script's own Inputs and Outputs docstring section, or from its hardcoded path constants where it has no such section.

#	Command	Writes	What it backs
1	bash scripts/fetch_data.sh	data/raw/{snli,multinli,chaosnli,med}/	Acquires the four raw datasets every other step reads.
2	python -m analysis.operator_tagger --input data/raw/snli/snli_1.0/snli_1.0_dev.jsonl --input-format jsonl --item-id-field pairID --premise-field sentence1 --hypothesis-field sentence2 --output data/derived/snli_dev_tagged.jsonl [b]	snli_dev_tagged.jsonl	Operator tagger output for SNLI dev; the monotonicity predictor for the confirmatory analysis.
3	Same command against multinli_1.0_dev_matched.jsonl, output data/derived/mnli_dev_matched_tagged.jsonl [b]	mnli_dev_matched_tagged.jsonl	Same, for MNLI dev matched.
4	Same command against multinli_1.0_dev_mismatched.jsonl, output data/derived/mnli_dev_mismatched_tagged.jsonl [b]	mnli_dev_mismatched_tagged.jsonl	Same, for MNLI dev mismatched, the secondary corpus.
5	python -m scripts.measure_agreement_marginals	stdout only	Marginal distribution of the agreement ordinal, measured before the predictor is touched.
6	python -m analysis.derive_power_rule_ordinal [d]	stdout only	Preregistered power analysis for the confirmatory contrast.
7	python -m scripts.match_chaosnli_overlap	chaosnli_overlap.json	ChaosNLI membership against SNLI and MNLI dev pairIDs. This is the bridge population and the Tier 3 sampling frame.
8	python -m scripts.measure_monotonicity_split	monotonicity_split.json	Marginal distribution of the predictor, measured before the predictor-outcome relation is computed. Needs 1 to 4.
9	python -m analysis.validate_med	med_validation.json	Validates the tagger's edit-site monotonicity direction against MED. Needs 1.
10	python -m analysis.dev_complexity	*_complexity.jsonl, dev_complexity_meta.json	Length and dependency-depth covariates for the confounder defense. Needs 1.
11	python -m analysis.confirmatory_ordinal	confirmatory_ordinal.json	The main confirmatory result, plus sensitivity analyses, the item-level ceiling, and the bridge. Needs 1, 2 to 4, 7, 8.
12	python -m analysis.confounder_defense	confounder_defense.json	Proportional-odds model checking whether the effect survives complexity and genre covariates. Needs 10 and 11's chain.
13	python -m analysis.tier2_flip_simulation	tier2_flip_simulation.json	Flip-sensitivity simulation for tagger misclassification. Needs 1 to 4, 8, 9.
14	python scripts/extract_tier3_sheet.py [c]	tier3_blind_sheet.tsv, tier3_key.json	Round 1 blind sample, 200 items, 50 per cell, and its key. Needs 1 to 4, 7.
15	python scripts/extract_tier3_resheet.py [c]	tier3_resheet.blind.tsv, tier3_resheet_key.json	Round 2 blind sample, drawn with round 1's 200 pairIDs excluded. Needs 14.
16	python -m scripts.rebuild_tier3_filled_sheets --output-dir data/manual [a]	data/manual/*.filled.tsv	Re-joins the published Tier 3 labels onto the regenerated text by item_no. Introduces no new labels or judgments. Needs 14, 15.
17	python -m analysis.tier3_agreement [a]	tier3_agreement.json	Round 1 agreement: four-class, binary collapse, kappa, per-cell, disagreement listing. Needs 16.
18	python -m analysis.tier3_resheet_agreement [a]	tier3_resheet_agreement.json	Round 2 agreement, plus disagreement-type counts. Needs 16.
19	python -m analysis.tier3_resheet_disagreements [a]	tier3_resheet_disagreements.md	Qualitative review of the round 2 disagreements, with the tagger re-applied for per-trigger detail. Needs 16, 18.
20	python -m analysis.appendix_tables	paper/appendix_tables.md, paper/latex/appendix_tables_{b,c,d}.tex	Appendix tables B1, B2, B3, C, D1, D2, as a readable markdown copy and as the LaTeX the paper includes. Needs 7, 11, 13, 18, 19.
21	python -m analysis.fig_forest	figures/fig_forest.{pdf,png}	Figure 1, the seven contrasts with bootstrap intervals and the SESOI band. Needs 11.
22	python -m analysis.fig_bridge_shares	figures/fig_bridge_shares.{pdf,png}	Figure 2, non-upward share by agreement level with Wilson intervals. Needs 11.

Steps 20 to 22 write outside data/derived/, as the table shows. Steps 5 and 6 write no file at all.

Footnotes:

• [a] Steps 17 to 19 read data/manual/*.filled.tsv at a hardcoded path, and those files carry corpus text, so they cannot ship. Step 16 rebuilds them from the published labels plus the text you regenerated in steps 14 and 15. Once step 16 has run, steps 17 to 19 read the same paths they always did and need no modification.
• [b] Reconstructed command. analysis/operator_tagger.py documents only a MED tagging example, so the three dev-corpus commands are assembled from evidence in other files: the --input-format jsonl branch is a real argparse-checked code path, the field names are confirmed by analysis/dev_complexity.py's docstring, and the output filenames are confirmed by analysis/confirmatory_ordinal.py's Inputs list. No single file states these commands verbatim. Treat the exact flags as a reconstruction to check rather than a quotation.
• [c] These two scripts document plain-script invocation rather than the -m form used elsewhere, and explain why in their own path-setup note: they insert the repository root onto sys.path themselves. The form shown is what the scripts specify.
• [d] analysis/derive_power_rule_ordinal.py reads no file at runtime. Its marginal counts are constants in the script, commented with the measurement provenance. Its position after step 5 reflects that provenance, not an enforced input.

analysis/compare_monotonicity.py ships as a library dependency only. Steps 11 to 15 import helper functions from it. It is not a step in this run order and cannot be run as a script here, because its own inputs belong to an earlier study and are not produced by this pipeline. Its docstring refers to design documents that live in that study's repository and are not included.

Paper to code map

The run order above reads forward, from command to output. This section reads backward, from a numbered item in the paper to the step that produces it, so a reviewer holding the paper can start anywhere.

The artifacts in the third column are not shipped. data/derived/ starts empty and each step fills it, so the column names the file that will carry the number once you have run that step, not a file you can open on a fresh clone.

Every table and figure in the paper:

Paper item	Produced by	Reading from
Table 1, all seven contrasts	step 11	confirmatory_ordinal.json
Tables 2 and 3, registration concordance	not generated; written by hand in the paper source	the registered plan, plus the results cited in each row
Table 4, ChaosNLI overlap arithmetic	step 20	chaosnli_overlap.json (step 7)
Table 5, ChaosNLI 100-label entropy in the overlap	step 20	confirmatory_ordinal.json (step 11)
Table 6, non-upward share by agreement level	step 20	confirmatory_ordinal.json (step 11)
Table 7, Tier 2 misclassification grid, 36 cells	step 20	tier2_flip_simulation.json (step 13)
Table 8, Tier 3 agreement and disagreement counts	step 20	tier3_resheet_agreement.json (step 18)
Table 9, all 25 Tier 3 disagreement items	step 20	tier3_resheet_disagreements.md (step 19) and tier3_resheet_agreement.json (step 18)
Figure 1, forest plot of the seven contrasts	step 21	confirmatory_ordinal.json (step 11)
Figure 2, non-upward share with Wilson intervals	step 22	confirmatory_ordinal.json (step 11)

Every class of statistic quoted in the running text:

Statistic in the paper	Produced by	Reading from
Analysis sample sizes and the two exclusion counts, 14 SNLI rows and 54 MNLI mismatched rows	step 11	confirmatory_ordinal.json, cross_checks
Cliff's delta, bootstrap intervals, Mann-Whitney p-values, Holm adjustment	step 11	confirmatory_ordinal.json
Sensitivity analyses A and B	step 11	confirmatory_ordinal.json, sensitivity
Item-level McFadden pseudo-R2	step 11	confirmatory_ordinal.json, item_level
Bridge figures for the ChaosNLI overlap, including 1,514 and 1,599	steps 7 and 11	chaosnli_overlap.json, confirmatory_ordinal.json, bridge
Proportional-odds result and the reported assumption violation	step 12	confounder_defense.json
Tagger validation against MED, including anchor rates	step 9	med_validation.json
Tier 2 flip-simulation rates quoted in the text	step 13	tier2_flip_simulation.json
Tier 3 round 1 agreement, kappa, per-cell breakdown	step 17	tier3_agreement.json
Tier 3 round 2 agreement, the four-class 0.875 quoted in the abstract	step 18	tier3_resheet_agreement.json
Predictor marginal distribution, the upward and non-upward split	step 8	monotonicity_split.json
Agreement ordinal marginals	step 5	stdout, no file
Power analysis behind the SESOI of 0.10	step 6	stdout, no file
Length and dependency-depth covariates	step 10	*_complexity.jsonl, dev_complexity_meta.json

Two entries above are deliberate exceptions and worth stating plainly. Tables 2 and 3 are not generated: they are a concordance between the registered plan and what happened, so each row is a claim about the study rather than a computed number, and the numeric claims inside them point at Table 1. Steps 5 and 6 print to stdout and write no file, so their numbers are reproduced by rerunning the step rather than by reading an artifact.

Determinism

Every step that uses randomness fixes a seed. The confirmatory bootstrap uses seed 20260717 with 10,000 percentile resamples. The Tier 3 samples use seed 20260717 with a fixed random-number consumption order. The Brant test validation uses seed 20260718. Steps 16 and 20 are fully deterministic and use no randomness.

License

Code in this repository is MIT licensed. See LICENSE.

The datasets are not covered by that license and are not distributed here. Each carries its own terms, listed in the data table above. ChaosNLI's non-commercial term in particular travels with the data, not with this code.