Skip to content

Commit 2b386a4

Browse files
GrituXGrituX
committed
[DOC] Adding test suite organization documentation
1 parent 1842114 commit 2b386a4

3 files changed

Lines changed: 208 additions & 0 deletions

File tree

‎CHANGELOG.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,4 @@ under development (0.3.2.dev0)
2929
- :bdg-success:`Feature` add leave-one-covariate-in (LOCI) method (:gh:`679` by `Marc Hulcelle`).
3030
- :bdg-danger:`Fix` fixed deprecated n_alphas with sklearn LassoCV, as well as deprecated penalty for LogisticRegressionCV (:gh:`690` by `Marc Hulcelle`)
3131
- :bdg-secondary:`Maint`: Fix conditional sampling test by verifying that sampler produces diverse samples. (:gh:`692` by `Joseph Paillard`_).
32+
- :bdg-primary:`Doc` add documentation about test suite organization and general logic. (:gh:`712` by `Marc Hulcelle`_).

‎tools/documentation_developer/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ You need more information, look here:
1313
- :ref:`Continuous Integration<developer_documentation_CI>`
1414
- :ref:`Building the documentation<developer_documentation_build>`
1515
- :ref:`Contributor's Code of Conduct <dev_code_of_conduct>`
16+
- :ref:`Test suite organization <test_suite>`
1617

1718
============
1819
CONTRIBUTORS
@@ -29,3 +30,4 @@ CONTRIBUTORS
2930
CI_documentation
3031
CONTRIBUTORS
3132
developer_code_of_conduct
33+
test_suite

‎tools/documentation_developer/test_suite.rst‎

Lines changed: 205 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,205 @@
1+
.. _test_suite:
2+
3+
===================================
4+
The Test Suite
5+
===================================
6+
7+
.. contents:: On this page
8+
:local:
9+
:depth: 2
10+
11+
Overview
12+
========
13+
14+
The test suite works under ``pytest``. The suite tests three different elements for every
15+
inference method:
16+
17+
- a correctness/statistical-power test on simulated data
18+
- an exceptions/error-handling test class, and, where the object wraps a scikit-learn estimator
19+
- a scikit-learn estimator guidelines compliance test.
20+
21+
We present here the structure of the test suite organization, the logic behind tests, and quickly
22+
present important fixtures, and other important utility functions.
23+
24+
Tests are automatically being run after each push on a pull-request (PR) whose target branch is ``main``. They are
25+
run on all combinations of python versions in ``[3.10, 3.14]``, and different OS structures
26+
``[Windows, Mac, Linux]``. We target a maximal code coverage with tests, meaning that tests should
27+
cover every line of existing code. To ensure this, we have the ``CodeCov`` utility bot that
28+
automatically computes the code coverage of tests after a successful commit on a PR, meaning that
29+
the entire test suite run is successful, and that the documentation generation process by ``Sphinx``
30+
is executed without any error. The code coverage metric should then appear after the execution of tests
31+
and documentation generation process directly on the PR discussion on `GitHub <https://github.com/mind-inria/hidimstat/pulls>`_.
32+
33+
How the Suite Is Organized
34+
============================
35+
36+
File Inventory
37+
--------------
38+
39+
.. list-table::
40+
:header-rows: 1
41+
:widths: 34 66
42+
43+
* - File
44+
- What it tests
45+
* - ``conftest.py``
46+
- Shared pytest fixtures and helpers (no tests of its own)
47+
* - ``__init__.py``
48+
- Empty — makes the test folder a package so relative imports
49+
(``from .conftest import ...``) work
50+
* - ``test_version.py``
51+
- Sanity check that ``hidimstat.__version__`` is a valid string
52+
* - ``test_base_perturbation.py``
53+
- ``BasePerturbation`` / ``BasePerturbationCV`` base classes shared
54+
by perturbation-style importance methods
55+
* - ``test_base_variable_importance.py``
56+
- ``BaseVariableImportance`` — the shared selection/plotting logic
57+
(top-k, percentile, threshold, FDR, FWER selection; plotting)
58+
* - ``test_permutation_feature_importance.py``
59+
- ``PFI`` / ``PFICV`` (Permutation Feature Importance)
60+
* - ``test_conditional_feature_importance.py``
61+
- ``CFI`` / ``CFICV`` (Conditional Feature Importance)
62+
* - ``test_leave_one_covariate_in.py``
63+
- ``LOCI`` / ``LOCICV`` (Leave-One-Covariate-In)
64+
* - ``test_leave_one_covariate_out.py``
65+
- ``LOCO`` / ``LOCOCV`` (Leave-One-Covariate-Out)
66+
* - ``test_desparsified_lasso.py``
67+
- ``DesparsifiedLasso``, the ``reid``/group-Reid noise estimators
68+
* - ``test_knockoff.py``
69+
- ``ModelXKnockoff``, ``GaussianKnockoffs`` (Model-X knockoffs)
70+
* - ``test_distilled_conditional_randomization_test.py``
71+
- ``D0CRT`` (distilled conditional randomization test / dCRT)
72+
* - ``test_ensemble_clustered_inference.py``
73+
- ``CluDL`` / ``EnCluDL`` (clustered and ensembled-clustered
74+
desparsified Lasso inference on spatial/temporal data)
75+
* - ``_utils/``
76+
- The data generation methods, and various method specific
77+
and non-specific utility functions.
78+
* - ``samplers/``
79+
- Conditional sampling functions, and samplers for Gaussian knockoffs.
80+
* - ``statistical_tools/``
81+
- Statistical tests functions for test aggregations, multiple testing,
82+
Nadeau-Bengio T-tests, and computations of p-values.
83+
84+
85+
The test suite follows the convention that there is a single test file for each single
86+
source python file. It ensures that each estimator or family of estimators has its own test
87+
module named after the corresponding source module. This keeps tests discoverable and
88+
lets a contributor working on one method run just ``pytest test_<method>.py``.
89+
90+
Three recurring layers of testing
91+
-----------------------------------
92+
93+
Across almost every file, tests fall into three categories:
94+
95+
- Statistical correctness on simulated data:
96+
Tests generate synthetic data with a known support (important vs.
97+
non-important features) via data simulation functions
98+
found in ``hidimstat._utils.scenario``, run the method, and
99+
check that:
100+
101+
- importance scores are higher on truly important features than on
102+
null features,
103+
- a selection procedure keeps the empirical false discovery
104+
proportion (FDP) below the target level and achieves reasonable
105+
statistical power.
106+
107+
Because these are stochastic tests, many are repeated over multiple
108+
seeds and the assertions compare an average FDP/power against a target plus a small tolerance, rather
109+
than requiring every single draw to succeed.
110+
111+
- API and exception behavior:
112+
Dedicated classes or functions check that
113+
invalid inputs raise the right error type with the expected message,
114+
that calling ``.importance()`` before ``.fit()`` fails clearly, and that
115+
warnings are raised as expected.
116+
117+
- Scikit-learn estimator compliance:
118+
As feature importance methods are written in compliance to scikit-learn standards,
119+
each one of them is checked against scikit-learn's own estimator conformance checks,
120+
so they behave like any other scikit-learn estimator.
121+
122+
Reproducibility testing
123+
-------------------------
124+
125+
Modules that can be seeded run the same four-way check on ``random_state`` handling:
126+
127+
- an integer seed gives identical results across repeated calls and
128+
across separate instances;
129+
- ``random_state=None`` gives different results on every call and
130+
every new instance;
131+
- a NumPy ``Generator`` object is consumed statefully. Repeated calls on the same
132+
instance differ, but re-creating the generator with the same seed reproduces the original
133+
result.
134+
135+
This is important for any Monte Carlo–style method where reproducibility guarantees matter.
136+
137+
Important fixtures and utilities
138+
=================================
139+
140+
Defined in the testing configuration file
141+
-----------------------------------------
142+
143+
``pytest_configure(config)``
144+
Forces matplotlib's non-interactive ``Agg`` backend so plotting
145+
tests never pop up a window during a test run.
146+
147+
``rng`` *(function-scoped)*
148+
Returns a freshly seeded ``numpy.random.default_rng(42)``. Used
149+
throughout the suite whenever a test needs its own controlled
150+
randomness (shuffling arrays, drawing p-values, generating toy
151+
datasets) independent of the main data-generation fixture below.
152+
153+
``data_generator`` *(function-scoped, parametrized)*
154+
The workhorse fixture of the whole suite. It depends on eight
155+
parameters — ``n_samples``, ``n_features``, ``support_size``,
156+
``rho``, ``seed``, ``value``, ``signal_noise_ratio``,
157+
``rho_serial`` — normally supplied via
158+
``@pytest.mark.parametrize`` on the test (or test class), and calls
159+
``multivariate_simulation`` to build a linear-model dataset with a
160+
known sparse support. It returns
161+
``(X, y, important_features, not_important_features)`` — the design
162+
matrix, the response, and the index arrays a test needs to check
163+
that a method ranks/selects the right features. Because the
164+
parametrize decorator sits on the *test*, different tests can reuse
165+
this one fixture with completely different problem sizes and noise
166+
regimes (e.g. high-dimensional ``n_features=200`` cases, correlated
167+
features via ``rho``, or noisy targets via ``signal_noise_ratio``).
168+
169+
``fitted_linear_regression()`` / ``_fitted_linear_regression()``
170+
A plain helper (not a fixture) that fits a ``LinearRegression`` on a
171+
tiny random 2-column dataset, used to seed estimator-check lists
172+
with an *already-fitted* model — several checks specifically probe
173+
fitted-vs-unfitted behavior.
174+
175+
``check_estimator(estimators, return_expected_failed_checks, valid=True)``
176+
The sklearn < 1.6 compatibility helper described above; not a
177+
fixture in the pytest sense, but a generator consumed by
178+
``@pytest.mark.parametrize`` to build ``(estimator, check, name)``
179+
tuples for valid or intentionally-invalid checks.
180+
181+
Local, file-specific fixtures
182+
--------------------------------
183+
184+
Several test files define their own narrower fixtures on top of the
185+
shared ones, generally to avoid refitting an expensive model in every
186+
test. Please make sure that you have checked module-scoped fixtures
187+
before implementing a utility function. Fixtures can generally be found
188+
at the beginning of the file.
189+
190+
Non-fixtures testing utilities
191+
------------------------------
192+
193+
If you ever need a non-fixture utility function, please make sure that you check existing functions in the separate
194+
``_utils``, ``samplers``, and ``statistical_tools`` modules before reimplementing a method.
195+
196+
Summary
197+
=========
198+
199+
The suite is thought to ensure having reusable testing
200+
recipes, applied consistently to every inference method in the library.
201+
The testing configuration file ``conftest.py`` supplies the two
202+
fixtures ``rng`` and ``data_generator``, and one helper
203+
``check_estimator`` that make this consistency possible, while each
204+
test module adds a thin, method-specific layer of fixtures and
205+
parametrizations on top.

0 commit comments

Comments
 (0)