feat(sleep): add correctness-first annotation case study

README.md

@@ -16,12 +16,12 @@ Benchmark case studies and shared benchmark tooling for HED (Hierarchical Event
 
 ## Case studies
 
-| Case study                    | Status  | What it measures                                                                      |
-| ----------------------------- | ------- | ------------------------------------------------------------------------------------- |
-| `use_cases/synthetic_search/` | working | Performance of the three hedtools HED search engines on synthetic and real event data |
-| sleep staging                 | planned | Assessment of HED-annotated sleep data                                                |
-| language scoring (lang)       | planned | Language scoring benchmarks                                                           |
-| epilepsy scoring (score)      | planned | Epilepsy scoring benchmarks based on SCORE                                            |
+| Case study                    | Status  | What it measures                                                                        |
+| ----------------------------- | ------- | --------------------------------------------------------------------------------------- |
+| `use_cases/synthetic_search/` | working | Performance of the three hedtools HED search engines on synthetic and real event data   |
+| `use_cases/sleep/`            | working | Correctness of source-specific semantic retrieval from parallel sleep-stage annotations |
+| language scoring (lang)       | planned | Language scoring benchmarks                                                             |
+| epilepsy scoring (score)      | planned | Epilepsy scoring benchmarks based on SCORE                                              |
 
 Each case study directory has its own README and the same committed layout: `src/` (scripts), `example/` with a small vendored test dataset (`test_data/`) and the results of running the benchmark on it (`test_data_results/` with `output/`, `figures/`, `reports/`), and `json_specifications/` (the case's standardized JSON specs, placeholder until the format lands). The scripts take `--data-dir` and `--results-dir` options to run on other datasets, whose results normally stay outside the repository.
 
@@ -41,6 +41,8 @@ uv pip install -e ".[dev,test]"
 ```
 python use_cases/synthetic_search/src/search_benchmark.py --quick
 python use_cases/synthetic_search/src/report.py
+python use_cases/sleep/src/sleep_case.py
+python use_cases/sleep/src/report.py
 ```
 
 ## Development

docs/use_cases/sleep.md

@@ -1,17 +1,21 @@
 ---
 myst:
   html_meta:
-    description: Sleep staging benchmark (planned) - assessment of HED-annotated sleep data
-    keywords: HED, benchmark, sleep, sleep staging, planned
+    description: Sleep annotation source search - correctness of source-specific HED retrieval
+    keywords: HED, benchmark, sleep, sleep staging, semantic search
 ---
 
-```{index} sleep staging, planned benchmark
+```{index} sleep staging, semantic search, source-specific annotations
 ```
 
-# Sleep staging benchmark
+# Sleep annotation source search
 
-**Status: planned.** No `use_cases/` directory exists yet.
+**Status: working.** The first sleep case checks source-specific semantic retrieval from parallel 30-second sleep-stage annotations.
 
-An application-oriented benchmark assessing HED-annotated sleep data. The concrete scope - datasets, annotation scheme, metrics, and reference results - is not yet defined; this page is the placeholder that will hold it.
+The committed example uses a compact synthetic fixture modeled on the CC0 [BOAS dataset](https://openneuro.org/datasets/ds005555) three-stream annotation structure. It asks where human consensus, PSG-based AI, and headband-based AI annotations contain N2 or an unavailable-data state, and which source produced each match.
 
-When work starts, the case study follows the repository conventions described in the {doc}`user guide <../user_guide>`: a `use_cases/sleep/` directory with its own README and the standard committed subdirectories (src/, example/test_data/, example/test_data_results/), and input/output in the standardized JSON format once that format lands.
+Expected rows and onsets come directly from the original numeric stage columns. They are therefore independent of the HED results used to test Basic, String, and Object search. Source-specific masks must be distinct, and broad semantic masks must be strict supersets.
+
+HED search is responsible for retrieving annotations by meaning while preserving source identity. Alignment, disagreement, transition context, agreement statistics, and scientific interpretation remain downstream analyses for a later real-data tutorial. No annotation source is treated as ground truth.
+
+See `use_cases/sleep/README.md` in the repository for commands, files, and the detailed correctness contract.

tests/test_sleep_case.py

@@ -0,0 +1,130 @@
+"""Tests for the sleep annotation source-search case."""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+
+import pytest
+
+REPO_ROOT = Path(__file__).parent.parent
+SRC_DIR = REPO_ROOT / "use_cases" / "sleep" / "src"
+DATA_DIR = REPO_ROOT / "use_cases" / "sleep" / "example" / "test_data"
+
+
+def _load_module(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    if spec is None or spec.loader is None:
+        raise ImportError(f"Cannot load {path}")
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+sleep_case = _load_module("sleep_case", SRC_DIR / "sleep_case.py")
+sleep_report = _load_module("sleep_report", SRC_DIR / "report.py")
+
+
+def _write_fixture_copy(target: Path, *, tsv_text: str | None = None, sidecar: dict | None = None) -> Path:
+    target.mkdir()
+    source_tsv = DATA_DIR / "sleep_annotation_events.tsv"
+    source_sidecar = DATA_DIR / "sleep_annotation_events.json"
+    (target / source_tsv.name).write_text(
+        tsv_text if tsv_text is not None else source_tsv.read_text(encoding="utf-8"),
+        encoding="utf-8",
+        newline="\n",
+    )
+    if sidecar is None:
+        sidecar = json.loads(source_sidecar.read_text(encoding="utf-8"))
+    (target / source_sidecar.name).write_text(
+        json.dumps(sidecar, indent=2) + "\n",
+        encoding="utf-8",
+        newline="\n",
+    )
+    return target
+
+
+def test_correctness_document_matches_independent_oracles():
+    document = sleep_case.build_correctness_document(DATA_DIR)
+    assert document["fixture"]["rows"] == 14
+    assert all(document["checks"].values())
+    expected = {
+        "any_n2": [30, 60, 90, 120, 150, 180, 210],
+        "human_n2": [30, 120, 150, 210],
+        "psg_ai_n2": [60, 120, 180, 210],
+        "headband_ai_n2": [90, 150, 180, 210],
+        "any_unavailable": [240, 270, 300, 330, 360],
+        "human_unavailable": [240, 330, 360],
+        "psg_ai_unavailable": [270, 330],
+        "headband_ai_unavailable": [300, 330, 360],
+    }
+    queries = {query["id"]: query for query in document["queries"]}
+    assert set(queries) == set(expected)
+    for query_id, onsets in expected.items():
+        assert queries[query_id]["expected"]["onsets_seconds"] == onsets
+        assert queries[query_id]["expected"]["count"] == len(onsets)
+        assert all(engine["correct"] for engine in queries[query_id]["engines"])
+
+
+def test_source_specific_oracles_are_distinct_strict_subsets():
+    frame, *_ = sleep_case.load_fixture(DATA_DIR, sleep_case.DEFAULT_SCHEMA_VERSION)
+    masks = sleep_case._direct_masks(frame)
+    for family in ("n2", "unavailable"):
+        broad = masks[f"any_{family}"].astype(bool)
+        specific = [masks[f"{source}_{family}"].astype(bool) for source in ("human", "psg_ai", "headband_ai")]
+        assert len({tuple(mask) for mask in specific}) == 3
+        for mask in specific:
+            assert int(mask.sum()) < int(broad.sum())
+            assert not bool((mask & ~broad).any())
+
+
+def test_correctness_and_report_are_deterministic(tmp_path):
+    first_results = tmp_path / "first"
+    second_results = tmp_path / "second"
+    first_path, _ = sleep_case.run_case(DATA_DIR, first_results)
+    second_path, _ = sleep_case.run_case(DATA_DIR, second_results)
+    assert first_path.read_bytes() == second_path.read_bytes()
+
+    first_report = sleep_report.generate_report(first_results)
+    second_report = sleep_report.generate_report(second_results)
+    assert first_report.read_bytes() == second_report.read_bytes()
+    text = first_report.read_text(encoding="utf-8")
+    assert text.startswith("# Sleep annotation search correctness\n")
+    assert text.count("## Query results") == 1
+    assert "No annotation source is treated as ground truth." in text
+
+
+def test_fixture_discovery_rejects_ambiguous_inputs(tmp_path):
+    with pytest.raises(ValueError, match="exactly one TSV"):
+        sleep_case._find_input_files(tmp_path)
+    (tmp_path / "events.tsv").write_text("onset\tduration\n", encoding="utf-8", newline="\n")
+    with pytest.raises(ValueError, match="Missing matching JSON sidecar"):
+        sleep_case._find_input_files(tmp_path)
+
+
+@pytest.mark.parametrize(
+    ("old", "new", "message"),
+    [
+        ("390\t30\t4\t4\t4", "390\t30\t99\t4\t4", "unsupported stage codes"),
+        ("0\t30\t0\t0\t0", "0\t25\t0\t0\t0", "durations must equal 30 seconds"),
+    ],
+)
+def test_fixture_contract_rejects_invalid_stage_or_duration(tmp_path, old, new, message):
+    text = (DATA_DIR / "sleep_annotation_events.tsv").read_text(encoding="utf-8")
+    variant = _write_fixture_copy(tmp_path / "variant", tsv_text=text.replace(old, new))
+    with pytest.raises(ValueError, match=message):
+        sleep_case.load_fixture(variant, sleep_case.DEFAULT_SCHEMA_VERSION)
+
+
+def test_fixture_validation_does_not_ignore_sidecar_warnings(tmp_path):
+    sidecar = json.loads((DATA_DIR / "sleep_annotation_events.json").read_text(encoding="utf-8"))
+    del sidecar["stage_hum"]["HED"]["4"]
+    variant = _write_fixture_copy(tmp_path / "missing_mapping", sidecar=sidecar)
+    with pytest.raises(ValueError, match="validation failed"):
+        sleep_case.load_fixture(variant, sleep_case.DEFAULT_SCHEMA_VERSION)
+
+
+def test_correctness_document_is_json_serializable():
+    document = sleep_case.build_correctness_document(DATA_DIR)
+    assert json.loads(json.dumps(document))["case_id"] == "sleep_annotation_source_search"

use_cases/sleep/README.md

@@ -0,0 +1,47 @@
+# Sleep annotation source search
+
+This case study checks whether HED search can retrieve parallel sleep-stage annotations by semantic meaning while preserving which scorer or algorithm produced each annotation.
+
+The first fixture is a synthetic 14-row table modeled on the CC0 [BOAS dataset](https://openneuro.org/datasets/ds005555) three-stream annotation structure. It contains human-consensus, PSG-algorithm, and headband-algorithm stage columns, but no participant data. The patterns are chosen so every source-specific oracle is distinct and every broad query is a strict superset of its source-specific queries.
+
+## Scientific question
+
+At which 30-second epochs do human consensus, PSG-based AI, and headband-based AI annotations contain N2 or an unavailable-data state, and which source produced each matching annotation?
+
+This is a semantic-search regression question. It does not treat any source as ground truth and does not estimate agreement, model performance, or population effects.
+
+## Correctness design
+
+Expected rows and onsets are calculated directly from the original numeric stage columns, without using HED output. Basic, String, and Object HED search must each reproduce those independent label-based oracles. The runner also checks that source-specific masks are distinct and that the broad semantic masks are strict supersets.
+
+Before HED expansion, the runner enforces the fixture contract: complete finite numeric fields, 30-second durations, unique ordered onsets, and only the documented human and AI stage codes. HED validation warnings are treated as failures rather than silently ignored.
+
+HED handles semantic retrieval. Alignment of parallel streams, disagreement, transition context, agreement statistics, and scientific interpretation remain downstream Python analyses for the later real-data tutorial.
+
+## Run the example
+
+From the repository root:
+
+```text
+python use_cases/sleep/src/sleep_case.py
+python use_cases/sleep/src/report.py
+```
+
+To use another compatible table and write results elsewhere:
+
+```text
+python use_cases/sleep/src/sleep_case.py --data-dir /path/to/data --results-dir /path/to/results
+python use_cases/sleep/src/report.py --results-dir /path/to/results
+```
+
+The data directory must contain exactly one TSV file and its same-stem JSON sidecar. The runner validates both against SCORE 2.1.0 by default. `--schema-version` overrides the schema explicitly.
+
+`sleep_correctness.json` is deterministic for fixed inputs and software behavior. Optional `--timing` writes a separate, timestamped local artifact with environment metadata. Those timings are machine-dependent orientation, not acceptance targets or cross-machine performance evidence.
+
+## Layout
+
+- `src/sleep_case.py` - validates the fixture and runs correctness-gated search.
+- `src/report.py` - renders the deterministic correctness report.
+- `example/test_data/` - synthetic stage table and HED sidecar.
+- `example/test_data_results/` - committed correctness output and report.
+- `json_specifications/` - fields learned from this case for later shared schema design.

use_cases/sleep/example/test_data/sleep_annotation_events.json

@@ -0,0 +1,67 @@
+{
+  "duration": {
+    "Description": "Duration of the scored interval.",
+    "Units": "s",
+    "HED": "Duration/# second"
+  },
+  "stage_hum": {
+    "LongName": "Human consensus sleep stage",
+    "Description": "Synthetic human consensus labels modeled on the CC0 BOAS annotation structure.",
+    "Levels": {
+      "0": "Wake",
+      "1": "Non-REM stage N1",
+      "2": "Non-REM stage N2",
+      "3": "Non-REM stage N3",
+      "4": "Rapid eye movement sleep",
+      "8": "Signal disconnection"
+    },
+    "HED": {
+      "0": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), (Experiment-participant, Awake)))",
+      "1": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), Sleep-stage-N1))",
+      "2": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), Sleep-stage-N2))",
+      "3": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), Sleep-stage-N3))",
+      "4": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), Sleep-stage-REM))",
+      "8": "({duration}, (Data-feature, (Human-agent, ID/human-consensus), Property-not-possible-to-determine, Label/signal-disconnection))"
+    }
+  },
+  "stage_psg_ai": {
+    "LongName": "Automated PSG sleep stage",
+    "Description": "Synthetic automated labels from polysomnography signals.",
+    "Levels": {
+      "-2": "Artifact or missing data",
+      "0": "Wake",
+      "1": "Non-REM stage N1",
+      "2": "Non-REM stage N2",
+      "3": "Non-REM stage N3",
+      "4": "Rapid eye movement sleep"
+    },
+    "HED": {
+      "-2": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), Property-not-possible-to-determine, Label/artifact-or-missing-data))",
+      "0": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), (Experiment-participant, Awake)))",
+      "1": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), Sleep-stage-N1))",
+      "2": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), Sleep-stage-N2))",
+      "3": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), Sleep-stage-N3))",
+      "4": "({duration}, (Data-feature, (Software-agent, ID/psg-ai), Sleep-stage-REM))"
+    }
+  },
+  "stage_headband_ai": {
+    "LongName": "Automated headband sleep stage",
+    "Description": "Synthetic automated labels from wearable EEG signals.",
+    "Levels": {
+      "-2": "Artifact or missing data",
+      "0": "Wake",
+      "1": "Non-REM stage N1",
+      "2": "Non-REM stage N2",
+      "3": "Non-REM stage N3",
+      "4": "Rapid eye movement sleep"
+    },
+    "HED": {
+      "-2": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), Property-not-possible-to-determine, Label/artifact-or-missing-data))",
+      "0": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), (Experiment-participant, Awake)))",
+      "1": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), Sleep-stage-N1))",
+      "2": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), Sleep-stage-N2))",
+      "3": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), Sleep-stage-N3))",
+      "4": "({duration}, (Data-feature, (Software-agent, ID/headband-ai), Sleep-stage-REM))"
+    }
+  }
+}

use_cases/sleep/example/test_data/sleep_annotation_events.tsv

@@ -0,0 +1,15 @@
+onset	duration	stage_hum	stage_psg_ai	stage_headband_ai
+0	30	0	0	0
+30	30	2	1	3
+60	30	1	2	3
+90	30	1	3	2
+120	30	2	2	3
+150	30	2	3	2
+180	30	1	2	2
+210	30	2	2	2
+240	30	8	1	3
+270	30	1	-2	3
+300	30	1	3	-2
+330	30	8	-2	-2
+360	30	8	3	-2
+390	30	4	4	4

use_cases/sleep/example/test_data_results/figures/README.md

@@ -0,0 +1,3 @@
+# Figures
+
+The first sleep case is a correctness regression and does not require a figure. Later real-data work may add figures for agreement, transition context, or interval relations after those analyses are defined independently of HED retrieval.