Add bootstrap confidence intervals and p-values to `rate_score()`

[aman-coder03]

current rate_score() returns point estimates only. Yadlowsky et al. (2021) derive a functional CLT enabling exact Gaussian inference via half-sample bootstrap, which is also implemented in the grf R package via rank_average_treatment_effect()

proposed extension is to add a return_ci=True parameter to rate_score()...

    rate_score(df, weighting="autoc", return_ci=True, n_bootstrap=200)

which would return the RATE estimate along with standard errors and p-values.

related to #887

[Whatsonyourmind]

The Yadlowsky et al. (2021) functional CLT result is the right foundation here. The key insight is that RATE is a functional of the CATE ranking, not just a point estimate, so standard bootstrap can be inconsistent — the half-sample bootstrap (or m-out-of-n bootstrap where m = n/2) is what gives valid coverage.

For implementation, the critical details are:

1. Resampling unit: bootstrap should resample observations, not CATE predictions. Each bootstrap draw re-estimates CATEs on the bootstrap sample, re-ranks, and recomputes the RATE integral. This captures both estimation uncertainty and ranking uncertainty.

2. Variance estimation: the Yadlowsky CLT gives sqrt(n) * (RATE_hat - RATE) -> N(0, sigma^2) where sigma^2 can be estimated via the half-sample bootstrap variance. The p-value tests H0: RATE = 0 (i.e., the model's prioritization is no better than random).

3. Computational cost: with n_bootstrap=200 and a heavy base learner (e.g., GBM), this gets expensive. A practical shortcut is to fix the nuisance model and only bootstrap the RATE functional over stored CATE predictions — this is what grf does and is valid under sample splitting.

The AUTOC weighting is the most natural default since it doesn't require a cost model, but the same bootstrap applies to QINI and TOC weightings.

[aman-coder03]

Thanks for the detailed notes! The point about bootstrapping observations rather than stored predictions is well taken, I'll make sure the implementation captures ranking uncertainty properly. The fast mode idea (fixing nuisance, bootstrapping only the RATE functional) also makes sense as a default for practical usability. Will incorporate this into the PR

[Whatsonyourmind]

@aman-coder03 Sounds like a solid plan. One thing to watch for in the implementation: when you bootstrap the RATE functional, the ranking can change between bootstrap samples — that's the whole point (capturing ranking uncertainty). But it means you need to re-sort inside each bootstrap iteration, which makes the fast mode ~O(n log n) per iteration instead of O(n). For n=10K observations and B=1000 bootstraps, that's still <1 second, but worth noting in case someone passes n=1M.

For the confidence interval output format, I'd suggest returning both the pointwise CI on the RATE curve and a scalar summary (like the area between the upper and lower CI bands). The scalar makes it easy to compare across experiments: "RATE CI width = 0.03" is more actionable than eyeballing overlapping bands.

Good luck with the PR — happy to review if you want a second pair of eyes on the bootstrap logic.

[aman-coder03]

good point on the re-sorting cost, I'll add a note about that in the docs. For the output format I'll check with the maintainers to keep it consistent with the rest of the API. Thanks!

[Whatsonyourmind]

@aman-coder03 Sounds good. The maintainer-consistency check is the right call — keeping the bootstrap CI output format aligned with the rest of the rate_score API will make it discoverable for users who already know the existing patterns.

One small note for the docs you're planning to add about the re-sorting cost: the dominant term is O(n log n) per bootstrap iteration from the sort itself, and the full bootstrap is O(B * n log n) for B bootstrap samples. For typical RATE applications with n ~ 10K and B ~ 1000, that's about 1.3e8 operations — fast enough to run in a few seconds on a single core. The naive concern would be "is this prohibitive" and the answer is "no, but it's not free either" — worth flagging the constant in case anyone wants to parallelize.

If you want to skip the per-bootstrap re-sort entirely, there's a trick using a precomputed rank transform: sort once on the original data, then for each bootstrap sample, compute the inverse-rank lookup in O(n) instead of re-sorting. Only works if the ranking is monotone in the bootstrap, which it is for the RATE functional. This drops the bootstrap complexity to O(B * n) which is 100x faster at n=10K. But it adds implementation complexity and the gain only matters at scale, so the docs note about the re-sort cost is enough for the v1 — the optimization can come later if anyone reports it as a bottleneck.

Good luck with the PR review.