Correct Bradley–Terry inference and make Rank recovery transactional

[AliFozooni]

Motivation

Rank previously updated the Bradley–Terry sufficient statistics with n_ij += pseudo and w_i += pseudo. Those two operations do not describe one coherent win matrix: they add denominator mass to every cell (including the diagonal and unobserved pairs), but only one numerator pseudo-win per item. Consequently, adding unrelated items could change an existing pair's contrast, disconnected groups were implicitly linked, and the reported inverse-information standard errors treated synthetic ridge information as observed data. A parsed draw also contributed two full comparisons.

Recovery had a second class of correctness risks. A resumed run could mix judgments after the input payload or effective judging specification changed, accept an incomplete or semantically malformed round, or cross a Batch submission/checkpoint gap that could duplicate paid work or lose completed rows.

This PR makes the estimator, uncertainty calculation, outcome semantics, and recovery protocol agree with one another.

Statistical model

For each observed undirected edge, the fitted directed win matrix is now

W_fit[i,j] = W_observed[i,j] + λ/2
W_fit[j,i] = W_observed[j,i] + λ/2

where λ = learning_rate. No mass is added to unobserved edges or the diagonal. For an isolated two-item edge this gives the exact contrast

s_i - s_j = log((w_ij + λ/2) / (w_ji + λ/2)).

This makes the perturbation symmetric, edge-local, and invariant to unrelated items.

Other inference changes:

• A draw is one fractional binary comparison: half a win in each direction. It is explicitly documented as a binary-BT working approximation, not a Davidson three-outcome tie model.
• insufficient_signal_policy="tie" makes an explicit equality assumption; "abstain" conditions the fit on retained judgments.
• learning_rate=0 now enforces Ford's directed strong-connectivity condition for every nontrivial observed component.
• Observed comparison components are fitted independently. Cross-component probabilities are undefined, isolates retain NaN scores/SEs, and z-scores are standardized only within identified components.
• Optimization combines a maximum-curvature empirical-logit spanning-tree start, a damped grounded-Laplacian Newton solve, Hunter MM fallback, and pairwise-stable SCC certification. Ill-conditioned directed strata are handled by a scaled Newton solve on the SCC condensation graph, followed by bounded coordinate polishing only when needed. This avoids losing regularization residuals hundreds of orders of magnitude below within-stratum curvature.

Uncertainty and diagnostics

For each observed component, standard errors use the fixed-graph model-based sandwich

Cov(s_hat) = B⁺ M B⁺

M = L(N_observed ⊙ p(1-p))
B = L((N_observed + λ·1_observed_edge) ⊙ p(1-p)),

under the component sum-to-zero constraint, where L is the graph Laplacian operator and B⁺ is the constrained pseudoinverse. Observed comparisons form the meat; fixed pseudo-comparisons add curvature only to the bread.

The documentation is explicit that this does not estimate shrinkage bias, shared-judge dependence, likelihood misspecification, adaptive-selection uncertainty, or uncertainty in the reported z-score. Non-recursive runs also write a diagnostics sidecar containing decoded outcome counts, effective comparison weight, graph components, isolates, and finite-SE counts.

Measurement and recovery integrity

• Pairwise responses must contain exactly the requested attributes and recognized outcomes before a round can commit.
• Normal rounds, catch-up work, recursive stages, and exact-tail retries now use durable intent/staging journals and atomic promotion.
• Input content fingerprints prevent reusing judgments after text or local-media payloads change.
• A secret-free measurement fingerprint covers the effective prompt, attributes, model, reasoning/search/media settings, endpoint, and judging configuration while excluding transport-only controls.
• Custom response_fn / get_all_responses_fn judges require an explicit judge_version, so hidden logic changes cannot silently mix measurement regimes.
• Batch submission intent is persisted before the irreversible create call; completed response rows are checkpointed before a Batch ID is removed. Ambiguous submissions fail closed.
• reset_files=True refuses to erase state that may refer to externally active Batch work.
• Run metadata and response checkpoints are atomically replaced.

The public API now applies the same raw-score projection to fresh and cached outputs, preserves unrelated user columns that happen to end in _raw / _se, rejects generated-column namespace collisions, and documents recursive output/identifiability behavior.

Compatibility

This intentionally changes Rank estimates that depended on the previous incoherent pseudo-count construction. Rank metadata is versioned, so incompatible historical checkpoints fail closed and require reset_files=True or a new save_dir. Custom judges additionally need a stable judge_version.

Validation

Committed regression coverage includes closed-form two-item estimates, irrelevant-item invariance, permutation/reversal equivariance, comparison to constrained SciPy likelihood optimization, fractional draws, Ford-condition failures, disconnected components and isolates, fixed-graph sandwich formulas, extreme separation, recursive identifiability, semantic response rejection, crash/retry boundaries, Batch intent recovery, reset safety, fingerprints, and cached API projection.

• pytest -q tests/test_rank_statistics.py tests/test_basic.py tests/test_reset_files.py tests/test_checkpoint_timeout_resume.py: 214 passed
• pytest -q tests/test_imports.py: 7 passed
• python -m build: wheel and source distribution built successfully
• Notebook JSON, py_compile, Ruff on the modified Rank/statistics paths, and git diff --check: passed
• Numerical stress harness:
  • all 4,374 / 4,374 complete four-item orientation/perturbation cases passed across six λ values from 1e-12 through 1e-300, with zero SCC-certification failures;
  • the separated-SCC analytic solution remained permutation-equivariant across all 24 item orders;
  • the deterministic 16-item / four-SCC λ≈1.65e-227 regression completes in about 0.24 s and certifies;
  • a deterministic 200-item, 1,400-edge, four-SCC case completes in about 0.23 s and certifies.

The full local suite reports 343 passed, 6 skipped, 12 failed. The same 12 failures occur on base commit 9caced9: they are unrelated tests/test_plot_utils.py failures because Matplotlib 3.11 removed matplotlib.cm.get_cmap.

References

• Ford (1957), “Solution of a Ranking Problem from Binary Comparisons”
• Hunter (2004), “MM Algorithms for Generalized Bradley–Terry Models”
• Davidson (1970), “On Extending the Bradley–Terry Model to Accommodate Ties”
• Yan (2016), “Ranking in the Generalized Bradley–Terry Models When the Strong Connection Condition Fails”
• Bong and Rinaldo (2022), “Generalized Results for the Existence and Consistency of the MLE in the Bradley–Terry–Luce Model”

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[AliFozooni]

I have read and agree to the GABRIEL CLA. Post the signature and recheck comments for me.

[AliFozooni]

I have read the CLA Document and I hereby sign the CLA

[AliFozooni]

recheck

[AliFozooni]

The required CLA signature is now posted above, and the CLA bot’s conversation message reports that all contributors have signed. The required check nevertheless remains red because the workflow fails before it can persist the signature:

Error occurred when creating the signed contributors file: Branch cla-signatures not found. Make sure the branch where signatures are stored is NOT protected.

The workflow’s path-to-document also currently points to the placeholder https://github.com/OWNER/REPO/blob/main/CLA.md rather than this repository’s CLA.md. Could a maintainer please create/configure the unprotected cla-signatures branch (or correct the workflow’s storage branch and document path), then rerun CLA Assistant? Thank you.