Skip to content

Commit 8675af4

Browse files
000specificclaude
000specific
and
claude
committed
annogroups: design spec + Phase 1 cross-compaction handoff
Research-notebook design spec (the four canonical types + locked design decisions) and the Phase 1 build handoff note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 2c47b97 commit 8675af4

2 files changed

Lines changed: 313 additions & 0 deletions

File tree

‎gigantic_project-COPYME/ai/ai_FYIs/annogroups-HANDOFF-2026june18-phase1_build.md‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
<!-- ============================================================================
2+
AI: Claude Code | Opus 4.8 (1M context) | 2026 June 18
3+
Human: Eric Edsinger
4+
Purpose: Cross-compaction handoff for the in-flight `annogroups` subproject
5+
build (Phase 1). Read this + the design spec before continuing.
6+
Scope: The new annogroups subproject build and its design context.
7+
============================================================================ -->
8+
9+
# HANDOFF — `annogroups` subproject, Phase 1 build (2026-06-18)
10+
11+
## 0. READ FIRST (in order)
12+
13+
1. **Design spec (authoritative, user-approved)**:
14+
`research_notebook/research_ai/subproject-annogroups/DESIGN-annogroups_subproject.md`
15+
2. This handoff (current build state + next steps).
16+
3. Conventions as always: `ai/ai_FYIs/gigantic_conventions.md`.
17+
18+
Session is rooted at **`GIGANTIC/`** (framework development per §16) — this is a
19+
new *framework* subproject, not a user addition.
20+
21+
## 1. What we're building
22+
23+
A new first-class **`subprojects/annogroups/`** subproject that produces
24+
**annogroups** as a reusable product. It replaces the old `single`/`combo`/`zero`
25+
scheme (which lived inside `annotations_X_ocl` Script 001).
26+
27+
**Four canonical annogroup types** (canonical GIGANTIC terms, like
28+
`phylogenetic_block`), computed **per source database** (pfam, gene3d, tmbed,
29+
signalp, deeploc, …):
30+
31+
| Type | Definition | Identifier |
32+
|---|---|---|
33+
| `annogroup_feature` | sequences sharing one feature (multi-membership) | `annogroup_<source>_<accession>` e.g. `annogroup_pfam_PF00001` |
34+
| `annogroup_combination` | sequences sharing the same **distinct set** of features (unordered, alphabetical key) | `annogroup_<source>_combination<NNNNN>` + map |
35+
| `annogroup_architecture` | sequences sharing the same **ordered** arrangement of positional features (N→C by start,stop) | `annogroup_<source>_architecture<NNNNN>` + map |
36+
| `annogroup_absent` | sequences (in the proteome universe) with **no** feature from the source | `annogroup_<source>_absent` |
37+
38+
**Locked design decisions** (do NOT re-litigate — user-approved):
39+
- A **feature** = `(source, annotation_identifier)` from annotations_hmms.
40+
- **Architecture grouping key = coord-FREE ordered accession pattern** (so homologs
41+
group); each sequence's **coordinate-tagged** architecture
42+
(`PF00001_start10_stop50,…`) is stored on its **membership row**, not the shared key.
43+
- **combination** = distinct set, **alphabetically** sorted (canonical key); copies ignored.
44+
- **absent** needs the full proteome universe (genomesDB STEP_4) — to enumerate
45+
complete membership, the source must have run over every species70 proteome.
46+
- **One parser plugin per source** (`ai/scripts/parsers/<source>.py`); the four-type
47+
construction is shared. New source = new parser, nothing else changes.
48+
- **How many types a source yields (3 or 4) is data-determined**, not a user knob
49+
(whole-protein sources like deeploc → no architecture).
50+
- Counter IDs (combination/architecture) assigned **deterministically** (sort keys, number).
51+
- All inter-subproject I/O via `output_to_input/` (§2) — no other path.
52+
53+
## 2. Build status — Phase 1 IN PROGRESS
54+
55+
### DONE (written + compile-clean)
56+
`subprojects/annogroups/`:
57+
- `BLOCK_build_annogroups/workflow-COPYME-build_annogroups/ai/scripts/`:
58+
- `utils_annogroups.py` — Feature namedtuple, ID formatters, parser-plugin contract, helpers
59+
- `001_ai-python-resolve_sources_and_universe.py` — discover parsers + resolve `sources:` + build proteome universe
60+
- `002_ai-python-build_annogroups.py` — generic builder: imports `parsers/<source>`, builds the 4 types + map + membership
61+
- `003_ai-python-validate_results.py` — fail-fast cross-checks (partition checks, absent∪annotated==universe, counts)
62+
- `004_ai-python-write_run_log.py` — §45 run log
63+
- `parsers/__init__.py`, `parsers/pfam.py` — first source parser
64+
- `START_HERE-user_config.yaml` — species70, sources: all, input paths, exec/slurm knobs
65+
- `output_to_input/.gitkeep`, `upload_to_server/.gitkeep`
66+
67+
### TESTED
68+
- **Script 001 runs clean**: parsers discovered = [pfam]; **proteome universe = 1,375,926 sequences**; sources manifest + universe written to `OUTPUT_pipeline/1-output/`. (~4 s.)
69+
70+
### NOT YET DONE (the next steps)
71+
1. **Smoke-test 002 + 003 on pfam** (the build + validation). Command pattern
72+
(run directly, fast iteration — same as integrator smoke tests):
73+
```bash
74+
cd subprojects/annogroups/BLOCK_build_annogroups/workflow-COPYME-build_annogroups
75+
OUT="$PWD/OUTPUT_pipeline" # 001 already ran; universe present
76+
python3 ai/scripts/002_ai-python-build_annogroups.py --source pfam --config START_HERE-user_config.yaml --output_dir "$OUT"
77+
python3 ai/scripts/003_ai-python-validate_results.py --source pfam --config START_HERE-user_config.yaml --output_dir "$OUT"
78+
```
79+
Watch: feature/combination/architecture/absent counts; validation PASS. Sanity:
80+
absent ≈ 1.38M − (pfam-annotated, ~?); annotated ≈ proteins with ≥1 pfam.
81+
⚠️ 002 holds the 1.38M universe + membership in memory — fine locally, but watch RSS.
82+
2. **Write the workflow machinery** (NOT yet written): `ai/main.nf`,
83+
`ai/nextflow.config`, `ai/conda_environment.yml` (env `aiG-annogroups-build_annogroups`,
84+
python>=3.11 + pyyaml + `nextflow>=23,<26`), `RUN-workflow.sh` (§29; copy/adapt the
85+
integrator's), `upload_manifest.tsv`, `INPUT_user/README.md`.
86+
- main.nf wiring: `resolve_sources_and_universe` (once) → read `1_ai-sources_manifest.tsv`
87+
via `splitCsv(header:true)` into a sources channel → `build_annogroups` (per source) →
88+
`validate_results` (per source) → collect → `write_run_log`. (Mirror the integrator
89+
main.nf; per-source fan-out like the integrator's per-structure fan-out.)
90+
3. **Run end-to-end via NextFlow** (local, env auto-creates `<26` nextflow; `export TMPDIR=/tmp`
91+
before slurm — see memory `reference_sanitize_tmpdir_before_slurm_submit`).
92+
4. **Docs** (§3/§42/§48/§56): subproject README+AI_GUIDE, BLOCK README+AI_GUIDE,
93+
workflow README+ai/AI_GUIDE, INPUT_user README. Plus `RUN-update_upload_to_server.sh`
94+
(thin wrapper — copy integrator's). Add annogroups to the project landing/AI_GUIDE if listed.
95+
5. **PRESENT Phase 1 to the user for review** before adding more parsers or touching downstream.
96+
97+
### LATER PHASES (after Phase 1 sign-off)
98+
- Add per-source parsers (gene3d, smart, cdd, superfamily = positional like pfam;
99+
tmbed/metapredict = positional segments/regions, feature-level "has-X"; signalp;
100+
deeploc = whole-protein → feature+combination+absent only, no architecture).
101+
Each parser knows its own annotations_hmms `output_to_input/` location:
102+
pfam/gene3d/etc → `BLOCK_interproscan_parsed/<db>/`; tmbed→`BLOCK_tmbed/`;
103+
signalp→`BLOCK_signalp/`; deeploc→`BLOCK_deeploc/`; metapredict→`BLOCK_metapredict/`.
104+
- **Restructure downstream (breaking, user-accepted)**: `annotations_X_ocl` Script 001
105+
→ *load* annogroups instead of *create*; integrator `BLOCK_annotations_X_orthogroups`
106+
→ consume from `annogroups/output_to_input/`. ⚠️ 4 types × many sources ≫ today's ~74k
107+
pfam single+combo → OCL `path_states` volume could explode; **estimate before full OCL re-run.**
108+
109+
## 3. Input data (verified on disk, 2026-06)
110+
111+
- **pfam features**: `annotations_hmms/output_to_input/BLOCK_interproscan_parsed/pfam/pfam-<phyloname>.tsv`
112+
(70 files = species70). Cols: `Protein_Identifier`, `Accession` (PF#####), `Match_Start`, `Match_End`.
113+
- **proteome universe**: `genomesDB/output_to_input/STEP_4-create_final_species_set/species70_gigantic_T1_proteomes/*.aa`
114+
(70 FASTAs; headers `>g_..._n_<phyloname>` = full GIGANTIC IDs; universe = 1,375,926 seqs).
115+
- **available sources** in the annotation DB: cdd, deeploc, funfam, gene3d, go, interproscan,
116+
metapredict, ncbifam, panther, pfam, prints, sfld, smart, superfamily, tmbed (+ signalp where present).
117+
- Reference for the per-source-parser pattern: `annotations_hmms/BLOCK_build_annotation_database`
118+
(001 discover + `parse_<tool>` per source). Reference BLOCK scaffold: the integrator BLOCK
119+
built earlier this session (`integrator/BLOCK_annotations_X_orthogroups/workflow-COPYME-*`).
120+
121+
## 4. Git state
122+
123+
- Branch `main`, last pushed commit **`634a372`** (def==accession reformat).
124+
- `subprojects/annogroups/` is **new + untracked** — NOT committed yet (commit after Phase 1
125+
review). `OUTPUT_pipeline/` + `__pycache__` are gitignored runtime; don't commit them.
126+
127+
## 5. Already DONE earlier this session (do NOT redo) — all pushed
128+
129+
- Built `integrator/BLOCK_annotations_X_orthogroups` (annogroups × orthogroups, non-bilaterian-
130+
metazoan focus); server publishing wired; integrator added to server allowlist.
131+
- Fixed a §39 canonical-RUN violation: OCL `BLOCK_annotations_X_ocl` server was serving stale
132+
RUN_01 (no defs) — switched to canonical RUN_02, removed stale manifest, re-published.
133+
- Reformatted `Annotation_Definitions` everywhere to **`definition ==accession`** (OCL formatter
134+
`utils_run_summary.format_annotation_definitions` + backfilled RUN_02's 945 files; integrator
135+
passes through). Server serves the new format. Commits `549faf2, 575a394, 10ebd2b, 583fdad, 634a372`.
136+
137+
## 6. Gotchas / conventions in play
138+
139+
- NextFlow env pin `>=23,<26` (system module is 26.x which breaks these workflows).
140+
- `export TMPDIR=/tmp` before any slurm submit (Claude TMPDIR footgun).
141+
- **Save Chat! is hand-captured** here (session rooted at `GIGANTIC/` = Pattern A; the official
142+
`ai/ai_scripts/003` only auto-discovers `…-COPYME` Pattern B). 14 sessions captured 2026-06-18
143+
incl. current `3aa4a745`.
144+
- annogroup_* are **canonical terminology** — use exactly: feature / combination / architecture / absent.

‎gigantic_project-COPYME/research_notebook/research_ai/subproject-annogroups/DESIGN-annogroups_subproject.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
<!-- ============================================================================
2+
AI: Claude Code | Opus 4.8 (1M context) | 2026 June 18
3+
Human: Eric Edsinger
4+
Purpose: Design spec for a new first-class `annogroups` subproject — sequences
5+
grouped by shared annotation features, per source database, across the
6+
four canonical annogroup types. Captures the agreed model; gated behind
7+
user sign-off before any build.
8+
Scope: The proposed annogroups subproject + the downstream restructuring it
9+
implies (annotations_X_ocl, integrator). Planning doc in the AI sandbox.
10+
Status: DRAFT — for sign-off.
11+
============================================================================ -->
12+
13+
# Design spec — `annogroups` subproject
14+
15+
## 1. Purpose
16+
17+
A new `subprojects/annogroups/` subproject produces **annogroups** as a
18+
first-class, reusable GIGANTIC product. Downstream subprojects
19+
(`annotations_X_ocl`, `integrator`, future consumers) consume the **same**
20+
annogroup IDs from it — one source of truth. This replaces the current
21+
`single`/`combo`/`zero` scheme (pfam-only, created inside OCL Script 001).
22+
23+
## 2. Canonical definitions
24+
25+
These four are **canonical GIGANTIC terms** — like `phylogenetic_block` or
26+
`phylogenetic_path`. An **annogroup** is a set of sequences sharing an annotation
27+
feature pattern, **always in reference to one source** annotation database. Every
28+
annogroup ID is prefixed `annogroup_<source>_…`. A **feature** is one
29+
`( source, annotation_identifier )` from the annotations_hmms annotation database.
30+
31+
### `annogroup_feature`
32+
- **Definition**: sequences that share a single specific feature from the source,
33+
regardless of what else they carry. A sequence with N features belongs to N
34+
feature-annogroups (multi-membership).
35+
- **Identifier**: `annogroup_<source>_<annotation_identifier>`
36+
e.g. `annogroup_pfam_PF00001`, `annogroup_tmbed_TM`.
37+
38+
### `annogroup_combination`
39+
- **Definition**: sequences that share the same **unordered, distinct set** of
40+
source features (copies ignored).
41+
- **Identifier**: `annogroup_<source>_combination<NNNNN>` (zero-padded counter).
42+
Map: ID → the feature set, listed **alphabetically**.
43+
e.g. `annogroup_superfamilies_combination00010 → SSF00010,SSF00055`.
44+
45+
### `annogroup_architecture`
46+
- **Definition**: sequences that share the same **ordered arrangement** of
47+
positional source features (N→C by start coordinate, copies kept).
48+
- **Identifier**: `annogroup_<source>_architecture<NNNNN>` (zero-padded counter).
49+
Map: ID → the ordered feature list **with coordinates**.
50+
e.g. `annogroup_pfam_architecture00001 →
51+
PF00001_start10_stop50,PF00001_start100_stop150,PF00011_start200_stop250`.
52+
53+
### `annogroup_absent`
54+
- **Definition**: sequences (across the species-set proteomes) that have **no**
55+
feature from the source.
56+
- **Identifier**: `annogroup_<source>_absent` e.g. `annogroup_pfam_absent`.
57+
58+
Counters (`combination`/`architecture`) are scoped per (source, type), zero-padded,
59+
and assigned **deterministically** (sort canonical keys, then number) so the same
60+
input always yields the same IDs. `feature` and `absent` need no map; their IDs
61+
are self-describing.
62+
63+
## 3. Worked example
64+
65+
A protein with `PF00001@10-50, PF00001@100-150, PF00011@200-250`:
66+
67+
| Type | This protein belongs to |
68+
|---|---|
69+
| feature | `annogroup_pfam_PF00001` **and** `annogroup_pfam_PF00011` |
70+
| combination | the group keyed by `{PF00001, PF00011}` (alphabetical, deduped) |
71+
| architecture | the group keyed by `PF00001_start10_stop50,PF00001_start100_stop150,PF00011_start200_stop250` (N→C order) |
72+
| absent | not absent for pfam (it has pfam features) |
73+
74+
## 4. How many types a source yields is **data-determined** (not a user option)
75+
76+
A source produces 3 or 4 of the canonical types depending on the kind of data its
77+
tool emits — never a user knob:
78+
79+
- **Positional, multi-feature** sources (pfam, gene3d, smart, cdd, superfamily,
80+
tmbed segments, metapredict regions, …) → all **4** types.
81+
- **Whole-protein / presence** sources (deeploc localization) → **3** (feature,
82+
combination, absent); architecture is undefined for position-less features
83+
and is simply not produced.
84+
85+
Each source's parser also decides its **feature granularity**, e.g.:
86+
- `tmbed`: `has-TM` at the feature level, with combination/architecture built from
87+
the per-segment data in the annotation DB.
88+
- `metapredict`: `has-IDR` at the feature level, with combination/architecture
89+
built from the per-region data in the annotation DB.
90+
91+
## 5. Membership universe & `absent`
92+
93+
Per-sequence absence is trivial (did the source annotate this sequence?). The
94+
**proteome matters only for completeness of enumeration**: to list every member
95+
of any annogroup (incl. `absent`), the source must have run over the **entire**
96+
species-set proteome set.
97+
98+
- **Universe** = all sequences in the species-set proteomes.
99+
- `annogroup_<S>_absent` = universe − (sequences with ≥1 S-feature).
100+
101+
The proteome sequence lists are read from upstream **`output_to_input/`** —
102+
`genomesDB/output_to_input/STEP_4-create_final_species_set/` — per GIGANTIC
103+
convention that all inter-subproject data flows through `output_to_input/`
104+
(§2; no other path is allowed).
105+
106+
## 6. Modularity — one parser per source
107+
108+
- A **common framework/driver** reads the YAML, inspects the annotation DB for
109+
available sources, dispatches to per-source parsers, then builds the canonical
110+
types + maps + membership uniformly and writes outputs.
111+
- **One Python parser per source database**. Each parser owns its source's feature
112+
semantics and returns a normalized per-sequence feature list
113+
`sequence_id → [ ( accession, start, stop, is_positional ), … ]`; the framework
114+
does everything else identically. **New source = new parser**, nothing else
115+
changes.
116+
117+
## 7. Configuration (`START_HERE-user_config.yaml`)
118+
119+
- `species_set` (e.g. species70) — drives the proteome universe.
120+
- `sources:` — `all` (every source present in the annotation DB) **or** an explicit
121+
subset; the subproject inspects the DB and validates the request.
122+
- (No annogroup-type knob — type count is data-determined per §4.)
123+
124+
## 8. Inputs / outputs (all via `output_to_input/`, §2)
125+
126+
**Inputs**
127+
- `annotations_hmms/output_to_input/BLOCK_build_annotation_database/` —
128+
standardized feature rows per source (`Sequence_Identifier, Database_Name,
129+
Annotation_Identifier, Domain_Start, Domain_Stop`).
130+
- `genomesDB/output_to_input/STEP_4-create_final_species_set/` — proteome
131+
sequence lists (the universe).
132+
133+
**Outputs** (`annogroups/output_to_input/`), per source —
134+
- annogroup **membership** (sequence → annogroup, all produced types),
135+
- annogroup **map** (combination/architecture ID → feature set/list + type + counts),
136+
- a **manifest** of sources + types produced.
137+
138+
Available sources in the live DB (2026-06): cdd, deeploc, funfam, gene3d, go,
139+
interproscan, metapredict, ncbifam, panther, pfam, prints, sfld, smart,
140+
superfamily, tmbed (+ signalp where present).
141+
142+
## 9. Downstream restructuring (breaking — accepted, "worth the effort")
143+
144+
- **`annotations_X_ocl`**: Script 001 stops *creating* annogroups and instead
145+
*loads* them from `annogroups/output_to_input/`; OCL runs per annogroup as today.
146+
⚠️ Four types × many sources ≫ today's pfam single+combo (~74k); OCL scale and
147+
the `path_states` matrices grow a lot — **estimate volume before a full re-run**.
148+
- **`integrator/BLOCK_annotations_X_orthogroups`**: consumes annogroup membership
149+
from `annogroups/` instead of from `annotations_X_ocl`; its single+combo logic
150+
generalizes to the canonical types.
151+
152+
## 10. GIGANTIC conventions
153+
154+
New `subprojects/annogroups/` as a BLOCK-type subproject (§41:
155+
`BLOCK_build_annogroups`, framework driver + per-source parser scripts under
156+
`ai/scripts/`). NextFlow workflow (per-source fan-out), unified `RUN-workflow.sh`
157+
(§29), `START_HERE-user_config.yaml`, fail-fast (§36), `write_run_log` (§45),
158+
`output_to_input/` (§2) + `upload_to_server/` (§38), full doc set (§3, §42, §48,
159+
§56), per-BLOCK conda env (§28), self-documenting TSV headers (§34).
160+
161+
## 11. Build phases (after sign-off)
162+
163+
1. Scaffold `subprojects/annogroups/` + framework driver + the **pfam** parser;
164+
validate the four canonical types end-to-end on one source.
165+
2. Add remaining per-source parsers (positional + whole-protein cases).
166+
3. Wire `output_to_input/`; estimate full-source volume.
167+
4. Restructure `annotations_X_ocl` Script 001 to consume annogroups.
168+
5. Restructure the integrator to consume annogroups.
169+
6. Re-run + re-publish the chain; update docs across all three subprojects.

0 commit comments

Comments
 (0)