OnlineMartingaleTest.compute_p_value (mapie/exchangeability_testing/martingales.py:497)
implements the smoothed conformal p-value as
n_greater: int = int(np.sum(history > current_conformity_score))
n_equal: int = int(np.sum(history == current_conformity_score))
return float((1.0 + n_greater + u * n_equal) / (n + 1.0))
The +1 belongs inside the tie term, not outside it. The textbook (smoothed)
conformal p-value is
p_t = (#{s_i > s_t} + U * (#{s_i = s_t} + 1)) / (n + 1)
Cheapest way to see it: on tie-free input, n_equal is 0, so u is
multiplied by 0 and drops out of the numerator entirely. The method's own
docstring says the p-value "is uniformly distributed on [0, 1]" via a
random tie-breaker U, but on tie-free input the output is bit-identical
across every random_state:
from mapie.exchangeability_testing.martingales import OnlineMartingaleTest
import numpy as np
history = np.array([0.30, 0.40, 0.50, 0.60, 0.70])
for seed in (1, 2, 3):
print(OnlineMartingaleTest(random_state=seed).compute_p_value(
current_conformity_score=0.55, conformity_score_history=history))
# 0.5, 0.5, 0.5 -- identical for every seed, no ties in the input
Two separate claims, because they resolve differently:
- The p-value itself stays valid and is even conservative (the shipped
value is always at least the textbook one, by exactly (1-U)/(n+1)).
sup_t[P(p<=t) - t] = 0 at every n and tie-block size I swept, using
exact enumeration over the n+1 equally likely exchangeable slots the
test point can occupy (fractions.Fraction, no Monte Carlo).
- The default exchangeability test built on that p-value does not hold
its level. The shipped p-value has E[p_t] = 1/2 + 1/(2(n+1)), a
persistent positive drift. Since the default jumper_martingale's
multiplier is 1 + jump_size*(p - 0.5), that drift turns the wealth
process into a strict submartingale, and Ville's inequality (the
guarantee behind test_level) no longer applies.
Measured on exchangeable i.i.d. continuous scores (so ties are a.s.
irrelevant to this half of the claim), identical streams under both forms,
default test_method="jumper_martingale", reject threshold 1/test_level = 20:
| p-value form |
P(sup M >= 20) over 300 independent streams |
shipped (1+n_greater+u*n_equal)/(n+1) |
0.103 (31/300) |
corrected (n_greater+u*(n_equal+1))/(n+1) |
0.033 (10/300) |
documented bound (test_level) |
0.050 |
Both docstrings need to change with the code: martingales.py:451's
.. math:: block currently states the buggy formula (so it agrees with the
code, not with the cited reference), and :469-470's "uniformly
distributed on [0, 1]" claim is the property this bug breaks. The cited
reference, Fedorova, Gammerman, Nouretdinov, Vovk (2012), "Plug-in
Martingales for Testing Exchangeability on-line", ICML, Algorithm 1 p. 3,
has the +1 inside the tie term.
Not a pure test-only issue: tests/exchangeability_testing/test_online_tests.py
has two tests that hardcode the current (buggy) output and go red under the
fix — one at the formula itself, one incidentally, because the current
non-tied case is forced to exactly p=0.5 and a downstream martingale
update depended on that coincidence. PR to follow with the fix, the
docstring correction, and updated/added tests (including a determinism
check on tie-free input, and an E[p]=1/2 check under exchangeability).
Separately: mapie/exchangeability_testing/permutations.py:363's
rank-starts-at-1 construction looks like the same missing addend at a
glance, but it is a different (permutation-null) p-value and is correct as
written; it does not need this fix.
OnlineMartingaleTest.compute_p_value(mapie/exchangeability_testing/martingales.py:497)implements the smoothed conformal p-value as
The
+1belongs inside the tie term, not outside it. The textbook (smoothed)conformal p-value is
Cheapest way to see it: on tie-free input,
n_equalis 0, souismultiplied by 0 and drops out of the numerator entirely. The method's own
docstring says the p-value "is uniformly distributed on
[0, 1]" via arandom tie-breaker
U, but on tie-free input the output is bit-identicalacross every
random_state:Two separate claims, because they resolve differently:
value is always at least the textbook one, by exactly
(1-U)/(n+1)).sup_t[P(p<=t) - t] = 0at everynand tie-block size I swept, usingexact enumeration over the
n+1equally likely exchangeable slots thetest point can occupy (
fractions.Fraction, no Monte Carlo).its level. The shipped p-value has
E[p_t] = 1/2 + 1/(2(n+1)), apersistent positive drift. Since the default
jumper_martingale'smultiplier is
1 + jump_size*(p - 0.5), that drift turns the wealthprocess into a strict submartingale, and Ville's inequality (the
guarantee behind
test_level) no longer applies.Measured on exchangeable i.i.d. continuous scores (so ties are a.s.
irrelevant to this half of the claim), identical streams under both forms,
default
test_method="jumper_martingale", reject threshold1/test_level = 20:P(sup M >= 20)over 300 independent streams(1+n_greater+u*n_equal)/(n+1)(n_greater+u*(n_equal+1))/(n+1)test_level)Both docstrings need to change with the code:
martingales.py:451's.. math::block currently states the buggy formula (so it agrees with thecode, not with the cited reference), and
:469-470's "uniformlydistributed on
[0, 1]" claim is the property this bug breaks. The citedreference, Fedorova, Gammerman, Nouretdinov, Vovk (2012), "Plug-in
Martingales for Testing Exchangeability on-line", ICML, Algorithm 1 p. 3,
has the
+1inside the tie term.Not a pure test-only issue:
tests/exchangeability_testing/test_online_tests.pyhas two tests that hardcode the current (buggy) output and go red under the
fix — one at the formula itself, one incidentally, because the current
non-tied case is forced to exactly
p=0.5and a downstream martingaleupdate depended on that coincidence. PR to follow with the fix, the
docstring correction, and updated/added tests (including a determinism
check on tie-free input, and an
E[p]=1/2check under exchangeability).Separately:
mapie/exchangeability_testing/permutations.py:363'srank-starts-at-1 construction looks like the same missing addend at a
glance, but it is a different (permutation-null) p-value and is correct as
written; it does not need this fix.