Skip to content

Commit b5a86e4

Browse files
turbomamclaude
turbomam
and
claude
committed
docs: comprehensive documentation for ID deprecation workflow
- docs/deprecation-workflow.md (new): step-by-step guide for human and agent contributors covering ID era history, how to deprecate a term, how to allocate new IDs, what the bootstrap tools do vs. what routine maintenance looks like, and a never-reuse checklist - CLAUDE.md: new "ID Allocation and Term Deprecation" section with the key invariants (deprecated.tsv is source of truth, research tools were one-time bootstrap, never reuse burned IDs) and a quick-reference table - docs/metpo_id_allocation_reference.md: updated header to clearly mark the document as historical reference and point at the three live sources of truth (deprecated.tsv, audit report, deprecation-workflow.md) - README.md: add deprecation-workflow.md to Key Documentation list Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 9b3dab3 commit b5a86e4

4 files changed

Lines changed: 236 additions & 7 deletions

File tree

‎CLAUDE.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,56 @@ When cleaning up or organizing the repository:
7676

7777
---
7878

79+
## ID Allocation and Term Deprecation
80+
81+
### The invariants — read these before touching any ID
82+
83+
1. **`src/templates/deprecated.tsv` is the source of truth** for all deprecated METPO terms.
84+
It is committed to the repo and merged into `metpo.owl` on every build. Do not treat it
85+
as a generated artifact — it is hand-maintained.
86+
87+
2. **Never reuse a burned ID.** An ID is burned if it appears in `deprecated.tsv`, in any
88+
BioPortal submission extract, or in any tagged release. Check before allocating.
89+
90+
3. **The research tools were a one-time bootstrap.** `generate-deprecated-template` and
91+
`audit-id-allocation` scanned BioPortal submissions and git history in March 2026 (PR #375)
92+
to produce the initial `deprecated.tsv`. Do not re-run `generate-deprecated-template` as
93+
part of routine maintenance — you will overwrite any hand-edits made since then. Run it
94+
only as a recovery tool if `deprecated.tsv` is believed to be corrupt or badly out of sync.
95+
96+
4. **The BioPortal submission extracts under `metadata/ontology/historical_submissions/`
97+
may eventually be removed.** If they are, `deprecated.tsv` still encodes everything that
98+
mattered in them.
99+
100+
### Deprecating a term (the normal workflow)
101+
102+
1. Remove the row from `src/templates/metpo_sheet.tsv` or `metpo-properties.tsv`.
103+
2. Add a row to `src/templates/deprecated.tsv` with `owl:deprecated true`, label
104+
`"obsolete <former label>"`, and an obsolescence reason (`IAO:0000226` or `OMO:0001000`).
105+
3. Commit both changes in the same commit.
106+
107+
Full details, column format, and examples: `docs/deprecation-workflow.md`.
108+
109+
### Allocating a new ID
110+
111+
```bash
112+
make audit-ids # regenerates reports/id-allocation-audit.md
113+
```
114+
115+
The report's "Safe Ranges" section gives the next available class and property IDs.
116+
Also run a grep sanity-check: `grep "METPO:<candidate>" src/templates/*.tsv src/templates/deprecated.tsv`
117+
118+
### Quick reference
119+
120+
| Task | Command |
121+
|------|---------|
122+
| See all burned IDs | `cat reports/id-allocation-audit.md` |
123+
| Regenerate audit report | `make audit-ids` |
124+
| Rebuild deprecated OWL component | `make -C src/ontology -f metpo.Makefile regenerate-deprecated` |
125+
| Verify an ID is safe to use | `grep "METPO:XXXXXXX" src/templates/*.tsv src/templates/deprecated.tsv reports/id-allocation-audit.md` |
126+
127+
---
128+
79129
## Ontology Development
80130

81131
### Build Commands

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,7 @@ make notebooks/non-ols-terms/D3O.tsv
234234
- **Ontology editing**: `src/ontology/README-editors.md`
235235
- **CLI reference**: `docs/cli-reference.md` - Complete guide to all 30+ command-line tools
236236
- **Literature mining**: `literature_mining/README.md`
237+
- **Deprecation workflow**: `docs/deprecation-workflow.md` — how to deprecate terms and allocate new IDs
237238
- **ID allocation and deprecation**: `reports/id-allocation-audit.md` — auto-generated list of all active and burned IDs, next safe IDs to allocate
238239
- **Historical analysis**: `metadata/ontology/historical_submissions/README.md`
239240
- **Database metadata**: `metadata/databases/README.md`

‎docs/deprecation-workflow.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
# METPO Term Deprecation Workflow
2+
3+
**Purpose:** How to deprecate terms and allocate new IDs in METPO, and how the
4+
deprecated-term machinery works.
5+
6+
---
7+
8+
## The Short Version
9+
10+
- `src/templates/deprecated.tsv` is the **authoritative list of all deprecated METPO terms**.
11+
It is committed to the repo and merged into `metpo.owl` on every release build.
12+
- **Never reuse a burned ID.** Check `reports/id-allocation-audit.md` for the full list.
13+
- When you deprecate a term, edit `deprecated.tsv` directly — do not re-run the research tools.
14+
- When you add a new term, use the next safe ID from the audit report.
15+
16+
---
17+
18+
## Background: Three Eras of METPO IDs
19+
20+
METPO has gone through three numbering schemes. All are burned and must never be reused.
21+
22+
| Era | Format | Range | Status |
23+
|-----|--------|-------|--------|
24+
| Era 1 | 6-digit with leading zeros | `000001`–`000274` | Abandoned (submissions 2–3) |
25+
| Era 2 | 7-digit with leading zeros | `0000001`–`0000351` | Abandoned (submissions 4–5) |
26+
| Era 3 | `1xxxxxx` classes, `2xxxxxx` properties | `1000001`+ / `2000001`+ | Current scheme |
27+
28+
Within Era 3, IDs `1000001`–`1000327` were used for literature-mining terms (derived from
29+
OntoGPT/IJSEM extraction) and retired in BioPortal submission 9. They must not be reused.
30+
31+
---
32+
33+
## Allocating New IDs
34+
35+
Run `make audit-ids` to regenerate `reports/id-allocation-audit.md` and get current
36+
"next safe ID" values. As of the last regeneration:
37+
38+
- **Next safe class ID:** `METPO:1005041` (highest active class + 1)
39+
- **Next safe property ID:** `METPO:2000735` (highest active property + 1)
40+
41+
Always use `make audit-ids` for current values — the numbers above go stale as terms are added.
42+
43+
---
44+
45+
## Deprecating a Term
46+
47+
When a term is removed from the active templates (`src/templates/metpo_sheet.tsv` or
48+
`src/templates/metpo-properties.tsv`), it must be added to `src/templates/deprecated.tsv`.
49+
50+
### Step 1 — Remove from the active template
51+
52+
Delete the row from `metpo_sheet.tsv` or `metpo-properties.tsv`. This is typically done
53+
by editing the upstream Google Sheet and running `make download-sheets`.
54+
55+
### Step 2 — Add a row to `src/templates/deprecated.tsv`
56+
57+
The file is a ROBOT template. Column layout:
58+
59+
```
60+
ID label TYPE deprecated obsolescence reason
61+
ID LABEL TYPE A owl:deprecated AI IAO:0000231
62+
METPO:XXXXXXX obsolete <former label> owl:Class true OMO:0001000
63+
```
64+
65+
Add one row per deprecated term. Use:
66+
- **label**: `obsolete ` + the former rdfs:label (e.g. `obsolete lyses`)
67+
- **TYPE**: `owl:Class` for classes, `owl:ObjectProperty` / `owl:DataProperty` /
68+
`owl:AnnotationProperty` for properties
69+
- **deprecated**: always `true`
70+
- **obsolescence reason**: use `OMO:0001000` (out of scope) or `IAO:0000226`
71+
(placeholder removed) — see the existing rows for precedent. When in doubt, use
72+
`IAO:0000226`.
73+
74+
### Step 3 — Commit both changes together
75+
76+
The removal from the active template and the addition to `deprecated.tsv` should be in
77+
the same commit so the ontology is never in a state where an ID simply vanishes.
78+
79+
### Step 4 — Verify
80+
81+
```bash
82+
# Check the deprecated template is valid ROBOT input
83+
uv run generate-deprecated-template --help # confirms the CLI is available
84+
85+
# Regenerate the audit report to confirm the ID now shows as burned
86+
make audit-ids
87+
grep "METPO:XXXXXXX" reports/id-allocation-audit.md
88+
```
89+
90+
---
91+
92+
## How `deprecated.tsv` Gets Into the Ontology
93+
94+
`src/ontology/metpo.Makefile` passes `deprecated.tsv` to ROBOT alongside the main
95+
templates when building `components/metpo_sheet.owl`:
96+
97+
```makefile
98+
components/metpo_sheet.owl: ... ../templates/deprecated.tsv
99+
$(ROBOT) template \
100+
--template ../templates/deprecated.tsv \
101+
...
102+
```
103+
104+
The resulting component is then merged into `metpo.owl` during `prepare_release`.
105+
Every deprecated term in the file becomes a real OWL entity with:
106+
- `owl:deprecated true`
107+
- `rdfs:label "obsolete ..."`
108+
- `IAO:0000231` (has obsolescence reason) annotation
109+
110+
---
111+
112+
## The Audit and Bootstrap Tools (Research Machinery)
113+
114+
Two CLI tools support this workflow:
115+
116+
### `audit-id-allocation` → `reports/id-allocation-audit.md`
117+
118+
```bash
119+
make audit-ids
120+
# or directly:
121+
uv run audit-id-allocation -o reports/id-allocation-audit.md
122+
```
123+
124+
Scans current templates, all BioPortal entity extracts under
125+
`metadata/ontology/historical_submissions/entity_extracts/`, and all tagged git releases.
126+
Produces a markdown report with active counts, burned ID lists, provenance chains, and
127+
next safe IDs. **Run this before allocating any new ID.**
128+
129+
### `generate-deprecated-template` → `src/templates/deprecated.tsv`
130+
131+
```bash
132+
make -C src/ontology -f metpo.Makefile regenerate-deprecated
133+
# or directly:
134+
uv run generate-deprecated-template -o src/templates/deprecated.tsv
135+
```
136+
137+
Regenerates `deprecated.tsv` from scratch by rerunning the same scan as the audit tool.
138+
**This is a bootstrap/recovery tool.** Under normal maintenance you should not need to
139+
run it — just edit `deprecated.tsv` by hand when deprecating a term. Run it only if you
140+
suspect `deprecated.tsv` has drifted from reality (e.g. after a large batch deprecation
141+
or if the file is corrupted).
142+
143+
### How `deprecated.tsv` was originally created
144+
145+
`generate-deprecated-template` was run once (March 2026, PR #375) after a comprehensive
146+
scan of all 9 BioPortal submissions and 13 tagged releases. It found 1,168 burned IDs
147+
across all three eras and wrote them into `deprecated.tsv` with labels recovered from
148+
the same historical sources. That file is now the source of truth — the scan tools exist
149+
as a safety net, not as a required step in the maintenance loop.
150+
151+
If the BioPortal submission extracts or the `generate-deprecated-template` script are
152+
ever removed, the methodology is documented in `docs/metpo_id_ranges_research.md` and
153+
the full provenance is in `reports/id-allocation-audit.md`.
154+
155+
---
156+
157+
## Never-Reuse Checklist
158+
159+
Before allocating any new ID, verify it does not appear in:
160+
161+
1. `src/templates/metpo_sheet.tsv` or `src/templates/metpo-properties.tsv`
162+
2. `src/templates/deprecated.tsv`
163+
3. `reports/id-allocation-audit.md`
164+
165+
```bash
166+
ID=1005041
167+
grep "$ID" src/templates/*.tsv src/templates/deprecated.tsv reports/id-allocation-audit.md
168+
```
169+
170+
If any of those grep hits, pick a different ID.

‎docs/metpo_id_allocation_reference.md‎

Lines changed: 15 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,20 @@
11
# METPO ID Allocation Reference
22

3-
> **This document is partially superseded.** For the current, machine-generated
4-
> allocation audit, run `make audit-ids` or see `reports/id-allocation-audit.md`.
5-
> The automated report scans all templates, BioPortal submissions, and tagged releases.
6-
> This document retains the block layout and policy sections as human-curated context.
7-
8-
**Date:** 2026-02-13 (last manual update); see `reports/id-allocation-audit.md` for current counts
9-
**Purpose:** Prevent ID reuse when adding new terms.
3+
> **This document is a historical reference.** The definitive source of truth is now:
4+
>
5+
> - **`src/templates/deprecated.tsv`** — committed ROBOT template of all deprecated terms
6+
> (1,168 burned IDs as of March 2026). Edit this file directly when deprecating terms.
7+
> - **`reports/id-allocation-audit.md`** — auto-generated by `make audit-ids`; gives active
8+
> counts, burned ID lists with provenance, and next safe IDs to allocate.
9+
> - **`docs/deprecation-workflow.md`** — step-by-step guide for deprecating terms and
10+
> allocating new IDs.
11+
>
12+
> This document retains the era descriptions, ID block layout, and gap analysis as
13+
> human-curated historical context. The counts and "highest ID" values here are stale —
14+
> use `make audit-ids` for current numbers.
15+
16+
**Date:** 2026-02-13 (last manual update)
17+
**Purpose:** Historical reference — era descriptions, block layout, and policy background.
1018

1119
---
1220

0 commit comments

Comments
 (0)