allow_infinite_bounds=True skips the calibration-size guard and returns a finite bound where none is valid

[shivamlalakiya]

Describe the bug

Setting allow_infinite_bounds=True skips the calibration-size guard entirely rather than changing what happens once it is passed. Where no finite bound is valid, the caller who explicitly asked to be given +inf is handed a finite interval built on the sample maximum, with no warning. The default path raises at the same configuration.

This is the same np.clip that #974 is about, reached by a different route, and I am filing it separately because the fix suggested in #974 covers it but the other candidate fix does not. #974's window is the asymmetric path, where get_effective_calibration_samples halves the count so _check_alpha_and_n_samples is not tight enough. Tightening that check — the second option offered in #974 — would leave this path untouched, because on this path the check never runs.

Mechanism

Three lines, on master @ 60031ac:

• mapie/regression/regression.py#L1735 — if not allow_infinite_bounds: wraps the _check_alpha_and_n_samples call. The flag removes the guard.
• mapie/utils.py#L816-L817 — alpha_cor = ceil(alpha_ref * (n_calib + 1)) / n_calib, then np.clip(alpha_cor, 0, 1).
• mapie/utils.py#L832 — if not (unbounded and _alpha >= 1).

The last line is the crux. _alpha iterates alpha_ref, the level before the finite-sample correction. Infeasibility is ceil(alpha_ref * (n+1)) > n, which occurs at alpha_ref well below 1. So in exactly the regime the unbounded flag exists for, _alpha >= 1 is false, the infinity branch does not fire, alpha_cor is clipped from above 1 down to 1, and nanquantile(..., 1.0, method="lower") returns max(scores).

The guard and the branch it guards are written against two different quantities: one against the corrected level, one against the uncorrected one.

To Reproduce

import warnings
import numpy as np
from sklearn.linear_model import LinearRegression
from mapie.regression import (SplitConformalRegressor, CrossConformalRegressor,
                              JackknifeAfterBootstrapRegressor)

warnings.simplefilter("ignore")
rng = np.random.default_rng(0)
beta = np.array([1.0, -2.0, 0.5])
X = rng.normal(size=(200, 3)); y = X @ beta + rng.normal(size=200)
Xc = rng.normal(size=(10, 3)); yc = Xc @ beta + rng.normal(size=10)
Xte = rng.normal(size=(2, 3))

builders = [
    ("SplitConformalRegressor",
     lambda: SplitConformalRegressor(estimator=LinearRegression(),
                                     confidence_level=0.95, prefit=False),
     lambda m: m.fit(X, y).conformalize(Xc, yc)),
    ("CrossConformalRegressor",
     lambda: CrossConformalRegressor(estimator=LinearRegression(),
                                     confidence_level=0.95, cv=5),
     lambda m: m.fit_conformalize(Xc, yc)),
    ("JackknifeAfterBootstrapRegressor",
     lambda: JackknifeAfterBootstrapRegressor(estimator=LinearRegression(),
                                              confidence_level=0.95, resampling=10),
     lambda m: m.fit_conformalize(Xc, yc)),
]
for name, build, fit in builders:
    for flag in (False, True):
        try:
            m = build(); fit(m)
            _, iv = m.predict_interval(Xte, allow_infinite_bounds=flag)
            w = float(iv[0, 1, 0] - iv[0, 0, 0])
            print(f"{name:34s} allow_infinite_bounds={str(flag):5s} -> width {w:.4f}  finite={np.isfinite(w)}")
        except Exception as e:
            print(f"{name:34s} allow_infinite_bounds={str(flag):5s} -> {type(e).__name__}: {str(e).splitlines()[0][:60]}")

At n_calib = 10, confidence_level = 0.95, the required rank is ceil(0.95 * 11) = 11 > 10, so no finite bound is valid. Output on 1.4.1:

SplitConformalRegressor            allow_infinite_bounds=False -> ValueError: Number of samples of the score is too low,
SplitConformalRegressor            allow_infinite_bounds=True  -> width 2.1261  finite=True
CrossConformalRegressor            allow_infinite_bounds=False -> ValueError: Number of samples of the score is too low,
CrossConformalRegressor            allow_infinite_bounds=True  -> width 2.8290  finite=True
JackknifeAfterBootstrapRegressor   allow_infinite_bounds=False -> ValueError: Number of samples of the score is too low,
JackknifeAfterBootstrapRegressor   allow_infinite_bounds=True  -> width 2.1830  finite=True

Three public regressor classes, same behaviour. TimeSeriesRegressor passes allow_infinite_bounds=True internally, so a caller who never touches the flag can also reach the path.

What it delivers

On a tie-free score set the returned threshold is the rank it landed on, and at every infeasible size that rank is n. So delivered coverage is n/(n+1), regardless of the level requested.

Enumerating the whole infeasible window in exact arithmetic — every n at which the required rank exceeds n, at two levels, with the first feasible size at each level kept as a control that must score zero:

requested	n	required rank	delivered	shortfall
0.95	2	3	0.6667	0.2833
0.95	10	11	0.9091	0.0409
0.95	18	19	0.9474	0.0026
0.95	19	19	0.9500	0.0000 (feasible — control)
0.90	8	9	0.8889	0.0111
0.90	9	9	0.9000	0.0000 (feasible — control)

Confirmed end to end through predict_interval on i.i.d. draws with a prefit base model, 50,000 replications per cell: 0.9078 ± 0.0013 at n=10, 0.95 against the exact 10/11 = 0.9091; no cell more than 1.19 s.e. from its exact value. Every interval returned was finite.

Direction: anti-conservative. Not a rounding loss — at these sizes no rank in the sample reaches the requested level at all.

Expected behavior

allow_infinite_bounds=True should return +inf here. That is what the parameter is documented to do, and it is the one case where the caller has explicitly said an infinite bound is acceptable.

Suggested fix

The fix suggested in #974 — have the quantile helper handle the infeasible case directly instead of clipping — resolves this too, and I think this issue is an argument for preferring it. Concretely, test the corrected level rather than the uncorrected one:

# mapie/utils.py, _compute_regression_quantile
alpha_cor_raw = np.ceil(alpha_ref * (n_calib + 1)) / n_calib
infeasible = alpha_cor_raw > 1
alpha_cor = np.clip(alpha_cor_raw, a_min=0, a_max=1)
# ... then branch on `infeasible[k]` rather than on `_alpha >= 1`:
#   unbounded  -> np.inf
#   otherwise  -> raise

That puts the decision next to the arithmetic that creates the condition, and it makes the guard at regression.py#L1735 redundant rather than load-bearing — which matters, because that guard is the thing the flag currently removes.

Happy to send a PR.

Versions

Behaviour verified on 1.4.1 (output above). The three lines are unchanged on master @ 60031ac, so I believe it is live there; I have not executed against master.

Disclosure

Found during an audit of level-to-rank conversion across conformal-prediction libraries. An earlier version of that work reported this clip as reachable, then retracted it: a scan over 49,990 combinations of calibration size, level and symmetry found no case where the guard passed and the clip bit. That scan held allow_infinite_bounds fixed at False and never varied it. Its own output said "dead code on this path", and the retraction generalised the qualifier away. This issue is the withdrawal of that retraction, with the flag varied.

[shivamlalakiya]

Two clarifications, so the reproducer and the permalinks line up.

The internal caller is a public method. I wrote that TimeSeriesRegressor passes the flag internally without naming where. It is TimeSeriesRegressor.adapt_conformal_inference, which calls self.predict(..., allow_infinite_bounds=True) inside its update loop. So the path is reachable through a public API by a caller who never touches the keyword.

Line numbers. The three references in the issue body are master @ 60031ac. The reproducer output is from 1.4.1, where the same code is at regression/regression.py:1714 and conformity_scores/interface.py:137-138 and :150.

No change to the report itself — both were verified against the sources cited.