Finalize evaluation module

[remrama]

A first draft has been merged, but some final upkeep needs to happen before yasa.evaluation is an official part of YASA.

• Review docstrings (especially the examples to make sure they did not fall behind code updates)
• Properly handle sleep statistics that are identical between scorers in SleepStatsAgreement
• Add unittests
• Update changelog
• Add tutorial with sample data
• Add SleepStatsAgreement.plot_blandaltman
• Add log-transform support to SleepStatsAgreement (Euser et al. 2008) #249

I think that all those tasks are required except the Bland-Altman plots, which might be technically optional though strongly preferred. Also, they should be implemented without Pinguoin, as we've managed to avoid that dependency everywhere else, plus the Bland-Altman plots here will need to have additional flexibility anyways (e.g., to accommodate the regression-based measures).

Sidenote: I'm still not sure what the best way to validate this pipeline is, as I'm not aware of an existing (public) dataset that has ground-truth values.

[remrama]

I've got the Bland-Altman and tutorial code on some local branches that I'll push and submit a PR for after #228 is merged. Holding on it just to avoid confusion since that PR has so much going on already.

[raphaelvallat]

Awesome, yes I agree, let's keep them separate PRs. I'm traveling for work all week with limited time for YASA, but hoping to review/answer your comments on #228 by next week 👍🏻

[raphaelvallat]

Just a side note: I got the approval from Luca Menghini and Fiona Baker to use the full dataset (sample_data.csv) from the Menghini standardized pipeline in YASA as well. Since this is the de facto standard & published reference, I think it's important that the YASA Evaluation module can be used to reproduce the main functions/results. Specifically, I'd like to create a new tutorial that replicates the original pipeline. With that said, as long as we have the option to replicate the original pipeline, I'm completely fine with having a different implementation logic, functions, or recommended default parameters than the original pipeline.

sample_data.csv

[remrama]

O cool! Great, I ended up finding some similar open datasets to use, but yes I agree it's much preferred to use this one. I will add this file to the YASA sample data Zenodo repository and use it for the evaluation module tutorial. Thanks for getting that from them.

[raphaelvallat]

@remrama re: full validation against the Menghini pipeline. I have asked Claude to extract all the references tables from the AnalyticalPipeline_v1.0.0.html (SRI-human-sleep/sleep-trackers-performance) file into a JSON file (attached below), and also generate Python tests of our evaluation pipeline against the reference tables. All tests pass ✅

Note: the tests currently only include per-subject and group means sensitivity/specificity

evaluation_sri_full.json

test_evaluation_sri.py

Files

evaluation_sri_full.json

Key	Source line	Content
per_subject	5505	Per-subject accuracy / sensitivity / specificity × 4 stages (WAKE, LIGHT, DEEP, REM) for all 14 subjects
per_subject_sleep_wake	5523	Per-subject accuracy / sensitivity / specificity for binary SLEEP/WAKE classification
group_ebe_staging	5845–5894	Group-level basic + advanced metrics (both avg with SD/CI across 14 subjects and pooled sum) for SLEEP STAGING
group_ebe_sleep_wake	5921–5966	Group-level basic + advanced metrics (both avg and pooled sum) for SLEEP/WAKE
error_matrices	5320–5362	Confusion matrices — proportional (mean ± SD [95% CI]) and absolute epoch counts — for both conditions
per_subject_sleep_measures	3057	Per-subject sleep architecture measures: TIB, TST, SE, SOL, WASO, Light / Deep / REM (minutes and %)
per_subject_discrepancies_staging	3187	Per-subject device − reference differences for all SLEEP STAGING measures
per_subject_discrepancies_sleep_wake	3201	Per-subject device − reference differences for binary SLEEP/WAKE measures

Regression tests (test_evaluation_sri.py)

sample_data.csv (all 14 subjects, 10 766 epochs) is loaded into YASA's EpochByEpochAgreement and the output is compared against this file.

Test class	JSON key used	What is checked	Tolerance
TestSRIPerSubject	per_subject[sbj][stage]["sensitivity"] / "specificity"	YASA recall and specificity × 100 match R pipeline values for every subject × stage combination (14 × 4 = 56 checks each), via subTest	0.1 pp
TestSRIGroupMeans	group_ebe_staging["basic_avg"][stage][metric]["mean"]	Arithmetic mean of YASA recall / specificity across all 14 subjects matches the R pipeline group average, per stage	0.5 pp
TestSRISanity	—	Dataset size (14 subjects), output shapes, index names, stage labels, all metrics in [0, 1], confusion matrix shape	—

Notes

• Subjects: sbj01–sbj14 (N = 14; all subjects from the HTML report).
• Stage encoding: 0 = Wake, 1 = Light (N1+N2), 2 = Deep (N3), 3 = REM; binary: 5 = Sleep.
• Units: EBE metrics in percentage points (0–100); sleep measures in minutes or %.
• avg vs sum: avg = mean (SD) [95% bootstrap CI] across subjects; sum = metrics computed on epochs pooled across all subjects.
• Advanced metrics (in group_ebe_*) include: PPV, NPV, Cohen's κ, PABAK, prevalence index.

Let me know if you want me to do a quick PR with these tests, or if you're already working on a tutorial and/or separate validation of the Menghini pipeline. Thanks!

[remrama]

Sweet, nice work. Awesome it's passing! Go ahead and submit this as a PR and request a review from me.

By the way, I was wondering if this line of #228 was going to be an issue:

 **Bug in standalone `groupDiscr.R:117`** (reference code, not YASA) | — | `logTransf=FALSE` should be `TRUE`. Bug in the R file, not YASA

But apparently not? 🤷 I'll keep that in mind during review.