upstream.py

#!/usr/bin/env python3
"""The schema, and the three modules needed to read a record against it.

`study-schema` is a submodule, not a copy, and the interface to it is deliberately
narrow -- the schema, and nothing that is merely code this layer happens to share:

    neuroimaging-study-extraction.yaml   what a record is
    storage-parameter-priorities.yaml    which of its fields matter most
    schema_utils                         reads those files: which slot is a wrapped
                                         value, a cross-reference, a nested object
    text_index                           the canonical normalization, the sha256
                                         that `ExtractionMetadata.source_text_hash`
                                         *is*, and the section index
    table_parse                          reads a coordinate table off the paper into
                                         rows, which is what a table *is* here

**The extraction schema, never the storage schema.** They are the same shape -- one
is a projection of the other -- but only the extraction side wraps a value in
`ExtractedValue`, and the wrapper is the whole point here: `extraction_status`,
`value_source` and `evidence.sets[].spans[]` are what a reviewer is shown and what
they correct. A record read against the storage schema would have values and no
warrant for any of them, and `classify_slot` would report every wrapped slot as a
nested object rather than as a field with evidence.

`storage-parameter-priorities.yaml` is not a schema, despite the name: it is a
`(class, field) -> rank` table, and it is read against extraction class names
because the extraction schema is a projection and keeps them. It resolves cleanly --
0 misses over the 609 value fields of the three baseline papers -- and it decides
two things: the `priority` a task carries for triage, and which of an entity's
fields go into the descriptor a reference is shown as.

The last three are there rather than vendored because they are not shared utilities:
they define the schema. `source_text_hash` and `EvidenceSpan.start_char` have no
meaning apart from the normalization `text_index` performs, so a second copy of it
would be a second definition of a schema field, and two parses of one table are two
disagreeing answers to which rows a paper reported.

Everything else this layer needs lives here. `tables.py` is the review grid --
attributing each parsed row to an analysis and rendering it. The one thing wanted
from `spans` was a twelve-line offset check, which is `staging.verify`. Extraction
itself is not here and not in the schema: it is pondie, which reads this same
submodule.

The dependency runs one way: the schema repo knows nothing about Label Studio, so
it can be checked out and used without this repo at all.

Importing this module is what puts the submodule on `sys.path`. One file knows the
layout; if the directory is ever renamed, this is the only place that changes.
"""

from __future__ import annotations

import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent

#: The submodule checkout. `git submodule update --init` fills it.
SUBMODULE = ROOT / "study-schema"

EXTRACTION_SCHEMA = SUBMODULE / "neuroimaging-study-extraction.yaml"
PRIORITIES = SUBMODULE / "storage-parameter-priorities.yaml"

#: The three baseline records the schema repo commits as worked examples. Test
#: fixtures, and nothing else: they are checked in there precisely so a downstream
#: consumer can prove itself against a record it did not invent. Production input
#: comes from `data/records`, not from here.
EXAMPLES = SUBMODULE / "examples"

_MISSING = """the study-schema submodule is empty at {path}.

    git submodule update --init

is what fills it. This layer reads the extraction schema, the priority inventory,
and the three modules that define what a record means -- schema_utils, text_index
and table_parse -- from there."""


def require() -> Path:
    """Put the submodule on `sys.path` and say something useful if it is not there."""

    if not EXTRACTION_SCHEMA.is_file() or not (SUBMODULE / "text_index.py").is_file():
        raise SystemExit(_MISSING.format(path=SUBMODULE))
    # Appended, never prepended. This repo's own modules have to win: the submodule
    # is a whole checkout, and a name it happens to share would otherwise shadow the
    # module actually being imported -- silently, and only for whoever has the
    # submodule at a commit where the collision exists.
    entry = str(SUBMODULE)
    if entry not in sys.path:
        sys.path.append(entry)
    return SUBMODULE


require()