Skip to content

OnlineMartingaleTest.compute_p_value: tie term costs the exchangeability test its level #984

Description

@shivamlalakiya

OnlineMartingaleTest.compute_p_value (mapie/exchangeability_testing/martingales.py:497)
implements the smoothed conformal p-value as

n_greater: int = int(np.sum(history > current_conformity_score))
n_equal: int  = int(np.sum(history == current_conformity_score))
return float((1.0 + n_greater + u * n_equal) / (n + 1.0))

The +1 belongs inside the tie term, not outside it. The textbook (smoothed)
conformal p-value is

p_t = (#{s_i > s_t} + U * (#{s_i = s_t} + 1)) / (n + 1)

Cheapest way to see it: on tie-free input, n_equal is 0, so u is
multiplied by 0 and drops out of the numerator entirely. The method's own
docstring says the p-value "is uniformly distributed on [0, 1]" via a
random tie-breaker U, but on tie-free input the output is bit-identical
across every random_state:

from mapie.exchangeability_testing.martingales import OnlineMartingaleTest
import numpy as np
history = np.array([0.30, 0.40, 0.50, 0.60, 0.70])
for seed in (1, 2, 3):
    print(OnlineMartingaleTest(random_state=seed).compute_p_value(
        current_conformity_score=0.55, conformity_score_history=history))
# 0.5, 0.5, 0.5 -- identical for every seed, no ties in the input

Two separate claims, because they resolve differently:

  1. The p-value itself stays valid and is even conservative (the shipped
    value is always at least the textbook one, by exactly (1-U)/(n+1)).
    sup_t[P(p<=t) - t] = 0 at every n and tie-block size I swept, using
    exact enumeration over the n+1 equally likely exchangeable slots the
    test point can occupy (fractions.Fraction, no Monte Carlo).
  2. The default exchangeability test built on that p-value does not hold
    its level.
    The shipped p-value has E[p_t] = 1/2 + 1/(2(n+1)), a
    persistent positive drift. Since the default jumper_martingale's
    multiplier is 1 + jump_size*(p - 0.5), that drift turns the wealth
    process into a strict submartingale, and Ville's inequality (the
    guarantee behind test_level) no longer applies.

Measured on exchangeable i.i.d. continuous scores (so ties are a.s.
irrelevant to this half of the claim), identical streams under both forms,
default test_method="jumper_martingale", reject threshold 1/test_level = 20:

p-value form P(sup M >= 20) over 300 independent streams
shipped (1+n_greater+u*n_equal)/(n+1) 0.103 (31/300)
corrected (n_greater+u*(n_equal+1))/(n+1) 0.033 (10/300)
documented bound (test_level) 0.050

Both docstrings need to change with the code: martingales.py:451's
.. math:: block currently states the buggy formula (so it agrees with the
code, not with the cited reference), and :469-470's "uniformly
distributed on [0, 1]" claim is the property this bug breaks. The cited
reference, Fedorova, Gammerman, Nouretdinov, Vovk (2012), "Plug-in
Martingales for Testing Exchangeability on-line", ICML, Algorithm 1 p. 3,
has the +1 inside the tie term.

Not a pure test-only issue: tests/exchangeability_testing/test_online_tests.py
has two tests that hardcode the current (buggy) output and go red under the
fix — one at the formula itself, one incidentally, because the current
non-tied case is forced to exactly p=0.5 and a downstream martingale
update depended on that coincidence. PR to follow with the fix, the
docstring correction, and updated/added tests (including a determinism
check on tie-free input, and an E[p]=1/2 check under exchangeability).

Separately: mapie/exchangeability_testing/permutations.py:363's
rank-starts-at-1 construction looks like the same missing addend at a
glance, but it is a different (permutation-null) p-value and is correct as
written; it does not need this fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions