Add option to include obsolete terms in reachable_from expansions

[cmungall]

Problem

When using a dynamic enum to validate term IDs in historical annotation data (e.g. reviewing existing GO annotations), the enum should accept both current and obsolete terms from the target ontology branch. Currently, reachable_from uses OAK's descendants(), which excludes obsolete terms because obsoletion removes the rdfs:subClassOf edge.

Concrete use case: ai4curation/ai-gene-review

Our schema has an ExistingAnnotation class representing GO annotations from the GOA database that a curator is reviewing. These may use terms that are now obsolete — the whole point of the review workflow is to mark such annotations for REMOVE or MODIFY action.

We'd like to bind a dynamic enum to ExistingAnnotation.term.id that:

• Accepts all current descendants of GO:0003674, GO:0008150, GO:0005575
• Also accepts obsolete GO terms (since historical annotations may reference them)

With the current schema, this isn't possible:

GOTermEnum:
  reachable_from:
    source_nodes: [GO:0003674, GO:0008150, GO:0005575]
    is_direct: false
    relationship_types: [rdfs:subClassOf]

returns ~40K current terms, but ~13K obsolete GO terms are silently excluded. Historical annotations using these get flagged as invalid.

For now we've removed the binding entirely (see ai4curation/ai-gene-review#249), but this loses a useful check — we can no longer verify that ExistingAnnotation.term.id is a valid GO CURIE at all.

Proposal

Add an optional include_obsoletes field (or similar name) to the reachable_from construct:

GOTermEnum:
  reachable_from:
    source_nodes: [GO:0003674, GO:0008150, GO:0005575]
    is_direct: false
    include_obsoletes: true
    relationship_types: [rdfs:subClassOf]

Implementation in _expand_reachable_from:

# Current (exclude obsoletes, via descendants())
descendants_result = adapter.descendants(source_node, predicates=predicates, reflexive=True)
values.update(descendants_result)

# New: also query obsolete terms with matching prefix if include_obsoletes
if getattr(query, "include_obsoletes", False):
    prefix = self._get_prefix(source_node)
    for curie, is_obs in adapter.entities(filter_obsoletes=False):
        if is_obs and curie.startswith(f"{prefix}:"):
            values.add(curie)

Or via a direct sql query if the adapter is a SqlImplementation:

SELECT DISTINCT subject FROM statements 
WHERE predicate = 'owl:deprecated'
AND subject LIKE 'GO:%'

Alternative: query replaced_by

Some projects may want "obsolete terms whose replacement is under this branch" rather than "all obsolete terms with this prefix". A more sophisticated form would be:

include_obsoletes: replaced_by_descendant  # only obsoletes whose term-replaced-by chain ends at a descendant of source_nodes

But a simple boolean covers the common case.

Why not use concepts?

We could technically list all ~13K obsolete GO terms as concepts, but:

1. That's fragile — needs re-generation when new terms are obsoleted
2. It requires serializing a huge enumeration in the schema
3. It defeats the purpose of dynamic enums (query at validation time)

Why not use matches with a regex?

matches matches on term syntax, not ontology membership. We'd match any GO:\d+-like string, not just legitimate obsolete terms under the CC/BP/MF roots.

Spec considerations

This would require a corresponding addition to the LinkML metamodel for ReachabilityQuery:

# In linkml meta schema
ReachabilityQuery:
  attributes:
    ...
    include_obsoletes:
      range: boolean
      description: >-
        If True, also include obsolete terms when expanding. Default False.
        Useful when validating historical annotation data that may reference
        deprecated terms.

Happy to open a corresponding linkml/linkml issue if this is better coordinated upstream first. But LTV is the consumer of the reachable_from semantics, so the behavior decision sits here.

Related

• fix: get validate-all to zero errors (549 → 0) ai4curation/ai-gene-review#249 — workaround: remove binding from ExistingAnnotation.term

[github-actions]

Summary

Problem Statement

This issue requests an include_obsoletes flag for the reachable_from dynamic enum construct, to support validation of historical annotation data that may legitimately reference obsolete ontology terms.

The root cause: OAK's descendants() only traverses live rdfs:subClassOf edges. When a GO term is deprecated, that edge is removed — so obsolete terms are invisible to reachable_from expansions. In GO alone, ~13K obsolete terms are currently excluded.

Concrete Driver: ai4curation/ai-gene-review

The ExistingAnnotation class in that schema represents GO annotations pulled from the GOA database. Curators review these to decide on REMOVE/MODIFY actions — which means by design, some annotations will reference obsolete GO terms. The validation schema binds ExistingAnnotation.term.id to a GOTermEnum built from reachable_from on the three GO roots (MF/BP/CC), but obsolete terms fail validation even though they're expected.

Workaround in place: ai4curation/ai-gene-review#249 removed the term.id binding entirely, losing the useful check that term.id is a valid GO CURIE at all.

Proposed Solution

Add include_obsoletes: bool (default False) to the reachable_from construct. When True, after the normal descendants() traversal, also fetch all entities with filter_obsoletes=False and include those matching the same CURIE prefix(es) as the source nodes.

Implementation hook: _expand_reachable_from() at src/linkml_term_validator/plugins/base.py:769

# After normal descendants traversal, if include_obsoletes=True:
if getattr(query, "include_obsoletes", False):
    for entity in adapter.entities(filter_obsoletes=False):
        curie = entity if isinstance(entity, str) else entity[0]
        if curie.startswith(f"{prefix}:") and adapter.is_obsolete(curie):
            values.add(curie)

Schema change also needed in the LinkML metamodel (ReachabilityQuery):

include_obsoletes:
  range: boolean
  description: >-
    If True, also include obsolete terms when expanding reachable_from.
    Useful for validating historical annotation data.

Alternative: replaced_by_descendant mode

A more targeted option: only accept obsolete terms whose replaced_by chain ends at a current descendant of the source nodes. Useful for stricter curation workflows. Could be a separate enum value for the field (include_obsoletes: replaced_by_descendant), but the simple boolean covers the most common case.

Why Not Other Approaches?

Approach	Problem
concepts: [...] listing all ~13K obsolete terms	Fragile; requires regeneration; bloats schema
matches: regex on prefix	Matches any GO:\d+-like string, not legitimate GO terms
Remove the binding entirely	Loses all CURIE-validity checking (current workaround in ai4curation/ai-gene-review)

Scope & Upstream Coordination

• This repo owns the behavior: _expand_reachable_from() expansion logic
• Upstream (linkml/linkml): Would need a corresponding include_obsoletes attribute added to the ReachabilityQuery metamodel class for schemas to express this cleanly. A companion issue in linkml/linkml is worth opening.
• OAK: Already supports filter_obsoletes=False in entities(), so no OAK changes needed.

Implementation Notes

• Cache key for _expand_reachable_from would need to incorporate include_obsoletes to avoid serving stale results
• Test data in tests/data/test_ontology.obo could add a deprecated term to cover the new code path
• The existing caching layer (in-memory + file) handles this naturally once the cache key is updated

Status

• Open, no PR yet
• Workaround in place in fix: get validate-all to zero errors (549 → 0) ai4curation/ai-gene-review#249 (binding removed)
• Implementation is straightforward; the main dependency is the upstream LinkML metamodel change for ReachabilityQuery.include_obsoletes