Skip to content

S-, T- and X-learner bootstrap confidence intervals are not reproducible #1036

Description

@jeongyoonlee

Depends on #1035 — that issue establishes the single check_random_state normalization path this one threads through the S- and X-learner families. Landing this first would mean either duplicating that decision or redoing it.

BaseLearner.bootstrap() (base.py:117) takes an optional rng and falls back to the global numpy stream when it is not supplied (base.py:133-137). #1032 fixed this for the R-learner (#1029). The remaining callers still pass no rng, so on master the R-learner is the only meta-learner whose bootstrap confidence intervals are reproducible:

DR-learner is unaffected: it threads an explicit per-call seed.

So BaseSRegressor(...).estimate_ate(..., bootstrap_ci=True) returns different intervals on every call, and unrelated consumption of the global numpy stream between calls changes the result. There is currently no way for a caller to pin them.

Note this is not purely a threading fix: grep -c random_state is 0 for both slearner.py and xlearner.py, so they have no random_state parameter to thread. Adding one is a backward-compatible API addition, which is why #1029 scoped itself to the R-learner.

Since #912 these are BaseEstimators, so the new parameter must be stored verbatim in __init__ per the scikit-learn convention, and each affected class wants clone() / get_params round-trip coverage. That is seven classes across two families: BaseSLearner, BaseSRegressor, BaseSClassifier, LRSRegressor (which constructs StatsmodelsOLS for the caller and needs care), plus BaseXLearner, BaseXRegressor, BaseXClassifier.

Suggested order: land #1035 first, then add random_state to BaseSLearner and BaseXLearner and pass rng at the six call sites.

Observed on master (477cd0a).

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions