annogroups: add 9 more source parsers (12 sources total), all validated end-to-end

Expands BLOCK_build_annogroups from 3 to 12 annotation sources (one parser plugin
each; sources: "all" auto-discovers them). Full DAG run on species70 validated all
12 PASS.

Positional (4 types: feature/combination/architecture/absent):
- gene3d, cdd, smart, superfamily, funfam: InterProScan-parsed positional DBs,
  clones of the panther/pfam parser (identical 15-col schema).
- tmbed: 3 segment classes (TM_helix / beta_barrel / signal_peptide) with coords;
  architecture = N->C membrane topology (e.g. 7-TM receptors group).
- metapredict: generic IDRs (single accession 'IDR'); feature/combination collapse
  to has-IDR, architecture groups by IDR count (user-accepted, low-resolution).

Whole-protein (3 types, no architecture):
- deeploc: subcellular-localization labels from the final-call 'Localizations'
  column (CSV); spaces/'/' in labels -> '_' accessions, readable label as definition.
- signalp: signal-peptide type (Sec/SPI etc.) from the SLOW model (user decision);
  '/' -> '_' accessions.

The four per-protein tools share one documented caveat: a protein not scored by the
tool falls into annogroup_<source>_absent ("no feature in the available output").

Docs (subproject + BLOCK + workflow READMEs/AI_GUIDEs + config) updated: source
inventory, per-source feature-type table, input locations, and the RUN_7 12-source
validation result (per-source annogroup counts).

The concurrent-session phyla->clades edits in composite_clades_manifest.tsv and
upload_manifest.tsv are intentionally left unstaged for that session.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: 000specific

gigantic_project-COPYME/subprojects/annogroups/AI_GUIDE.md

@@ -91,13 +91,19 @@ BLOCK guide.
 Built 2026-06-18. **pfam validated end-to-end** (137,762 annogroups across the
 four types; validation PASS; full NextFlow DAG run).
 
-2026-06-28: two more parsers added and validated (scripts 001–003, PASS):
-- **panther** — positional PTHR families (4 types): feature 11,033 / combination
-  11,033 / architecture 11,051 / absent 418,016.
-- **go** — whole-protein Gene Ontology terms from the **raw** InterProScan results
-  (3 types, no architecture by design): feature 8,994 / combination 27,974 / absent
-  539,984. GO origin selection (InterPro / PANTHER union, default both) is a
-  documented config knob (`go_term_origins`).
+2026-06-28: expanded to **12 sources**, all validated end-to-end (full DAG,
+species70, validation PASS):
+- positional (4 types): **pfam, panther, gene3d, cdd, smart, superfamily, funfam**
+  (InterProScan domains/families), **tmbed** (membrane topology — architecture is
+  the TM/barrel/signal segment order), **metapredict** (IDRs — one accession `IDR`,
+  so architecture groups by IDR count).
+- whole-protein (3 types, no architecture): **go** (GO terms; names from the
+  go-basic.obo mapping; origins via the `go_term_origins` knob, default InterPro +
+  PANTHER union; plus 6 GO-aspect split columns), **deeploc** (subcellular
+  localization), **signalp** (signal-peptide type, SLOW model).
+- The per-protein tools (tmbed/signalp/deeploc/metapredict) carry one caveat:
+  proteins not scored by the tool fall into `annogroup_<source>_absent` (absent =
+  "no feature in the available output").
 
 One known, user-accepted data caveat: a handful of truncated multi-locus annotation
 IDs are dropped — see

gigantic_project-COPYME/subprojects/annogroups/BLOCK_build_annogroups/AI_GUIDE.md

@@ -109,12 +109,21 @@ Conventions a parser must follow:
   WARNING note).
 
 `parsers/pfam.py` is the reference implementation (positional domains, all four
-types). **Implemented parsers**: `pfam` + `panther` (both
-`BLOCK_interproscan_parsed/<db>/`, positional, 4 types) and `go` (see below).
-Per-source input locations for further parsers:
-`gene3d/cdd/smart/superfamily/funfam` → `BLOCK_interproscan_parsed/<db>/`;
-`tmbed` → `BLOCK_tmbed/`; `signalp` → `BLOCK_signalp/`; `deeploc` →
-`BLOCK_deeploc/`; `metapredict` → `BLOCK_metapredict/`.
+types). **All 12 sources are implemented.** Input locations (all under the config
+`annotations_hmms_dir` root):
+
+- `pfam, panther, gene3d, cdd, smart, superfamily, funfam` → `BLOCK_interproscan_parsed/<db>/<db>-*.tsv` (15-col schema; positional clones)
+- `go` → raw `BLOCK_interproscan/*_interproscan_results.tsv` (see below)
+- `tmbed` → `BLOCK_tmbed/*_tmbed_predictions.tsv` (3 positional segment classes)
+- `signalp` → `BLOCK_signalp/*_signalp_SLOW_predictions.tsv` (whole-protein SP type; SLOW model)
+- `deeploc` → `BLOCK_deeploc/*_deeploc_predictions.csv` (whole-protein localization labels; CSV)
+- `metapredict` → `BLOCK_metapredict/*_metapredict_predictions.tsv` (generic positional IDRs)
+
+The per-protein tools (tmbed/signalp/deeploc/metapredict) share one caveat: a
+protein not scored by the tool (e.g. dropped upstream by the long-header filter)
+has no feature and so falls into `annogroup_<source>_absent` — i.e. `absent` means
+"no feature in the available output", which can include a few not-evaluated
+proteins (analogous to the dropped-orphan caveat).
 
 **`parsers/go.py` (special).** GO is not a standalone parsed database. The go
 parser reads the **raw** per-species results (`BLOCK_interproscan/*_interproscan_results.tsv`,
@@ -147,11 +156,14 @@ and 6-output (composite-clade per-annogroup + detail tables). Sources without
 (the N→C *ordered* arrangement, needs coordinates). Non-positional sources yield
 no architecture (3 types).
 
-| Source kind | Example | is_positional | Types |
+All 12 sources below are **implemented** (one parser plugin each; `sources: "all"`
+builds them all).
+
+| Source kind | Sources | is_positional | Types |
 |-------------|---------|---------------|-------|
-| positional domains/families | **pfam**, **panther** (implemented); gene3d, cdd, smart, superfamily, funfam | True | feature + combination + architecture + absent |
-| positional segments/regions | tmbed (TM segments), metapredict (IDRs), signalp (cleavage region) | True | all four (architecture = segment order) |
-| whole-protein label | **go** (Gene Ontology terms, implemented); deeploc (localization) | False | feature + combination + absent (no architecture) |
+| positional domains/families | pfam, panther, gene3d, cdd, smart, superfamily, funfam | True | feature + combination + architecture + absent |
+| positional segments/regions | tmbed (TM_helix / beta_barrel / signal_peptide; architecture = membrane topology), metapredict (IDRs; one accession `IDR`, so architecture = IDR count) | True | all four |
+| whole-protein label | go (GO terms), deeploc (subcellular localization), signalp (signal-peptide type, SLOW model) | False | feature + combination + absent (no architecture) |
 
 ## Validation (Script 003, fail-fast)
 

gigantic_project-COPYME/subprojects/annogroups/BLOCK_build_annogroups/README.md

@@ -77,13 +77,29 @@ Built 2026-06-18 — scaffold, scripts, docs, and the NextFlow workflow complete
 10,635 feature + 46,846 combination + 80,280 architecture + 1 absent; validation
 PASS).
 
-**2026-06-28: panther + go parsers added and validated** (build + validate,
-scripts 001–003, species70, validation PASS):
-- **panther** (positional, 4 types): 11,033 feature + 11,033 combination + 11,051
-  architecture + 1 absent (418,016 absent sequences).
-- **go** (whole-protein, 3 types — no architecture): 8,994 feature + 27,974
-  combination + 1 absent (539,984 absent sequences); GO origins = InterPro + PANTHER
-  union (default `go_term_origins`).
+**2026-06-28: expanded to 12 sources, all validated end-to-end** (full DAG,
+species70, validation PASS — universe 1,375,926 sequences). Total annogroups per
+source (feature / combination / architecture):
+
+| Source | annot% | feature | combination | architecture |
+|--------|-------:|--------:|------------:|-------------:|
+| pfam | 67.0 | 10,635 | 46,846 | 80,280 |
+| panther | 69.6 | 11,033 | 11,033 | 11,051 |
+| gene3d | 61.1 | 3,289 | 25,748 | 50,886 |
+| cdd | 28.5 | 11,239 | 18,657 | 26,037 |
+| smart | 31.4 | 1,171 | 10,974 | 35,042 |
+| superfamily | 58.4 | 1,412 | 18,386 | 37,579 |
+| funfam | 23.1 | 36,784 | 51,917 | 56,422 |
+| go | 60.8 | 8,994 | 27,974 | — |
+| deeploc | 100.0 | 10 | 177 | — |
+| signalp | 12.0 | 1 | 1 | — |
+| tmbed | 29.4 | 3 | 7 | 122 |
+| metapredict | 78.7 | 1 | 1 | 85 |
+
+(`—` = whole-protein source, no architecture. signalp/metapredict feature counts are
+small by nature: one signal-peptide type called in species70, one generic `IDR`
+accession; tmbed's 3 = TM_helix / beta_barrel / signal_peptide. go carries 6
+GO-aspect split columns.)
 
 One known, user-accepted caveat: truncated multi-locus annotation IDs are
 dropped — see [`workflow-COPYME-build_annogroups/ai/ai_FYIs/WARNING-truncated_orphan_annotations.md`](workflow-COPYME-build_annogroups/ai/ai_FYIs/WARNING-truncated_orphan_annotations.md).

gigantic_project-COPYME/subprojects/annogroups/BLOCK_build_annogroups/workflow-COPYME-build_annogroups/README.md

@@ -74,13 +74,15 @@ pfam-annotated = 922,233; **137,762 annogroups** (10,635 feature + 46,846
 combination + 80,280 architecture + 1 absent holding 453,693 sequences); validation
 **PASS**; 7 truncated orphan IDs dropped (audited).
 
-**panther + go** (2026-06-28, build + validate, scripts 001–003): same universe.
-- panther (positional, 4 types): 11,033 feature + 11,033 combination + 11,051
-  architecture + 1 absent (418,016 seqs); validation **PASS**; 7 orphans dropped.
-- go (whole-protein, 3 types — no architecture): 8,994 feature + 27,974 combination
-  + 1 absent (539,984 seqs); validation **PASS**; 6 orphans dropped; GO origins =
-  InterPro + PANTHER (default union, `go_term_origins`). GO build ≈ 45 s (reads the
-  raw per-species InterProScan results).
+**12 sources** (2026-06-28, full DAG, SLURM, ~11 min): same universe; all 12
+validated **PASS** — pfam, panther, gene3d, cdd, smart, superfamily, funfam (IPS
+positional → 4 types), tmbed + metapredict (positional segments → 4 types), go,
+deeploc, signalp (whole-protein → 3 types). See the per-source annogroup-count
+table in [`../README.md`](../README.md). New since the pfam-only run:
+- `go` GO term names + 6 GO-aspect (MF/BP/CC) split columns; origins via `go_term_origins`.
+- `tmbed` architecture = membrane topology; `deeploc` = subcellular localization
+  (100% annotated); `signalp` = signal-peptide type (SLOW model); `metapredict`
+  architecture = IDR count.
 
 See [`ai/AI_GUIDE.md`](ai/AI_GUIDE.md) for execution detail and troubleshooting,
 and [`../AI_GUIDE.md`](../AI_GUIDE.md) for the parser contract.

gigantic_project-COPYME/subprojects/annogroups/BLOCK_build_annogroups/workflow-COPYME-build_annogroups/START_HERE-user_config.yaml

@@ -22,9 +22,10 @@ species_set_name: "species70"
 # Which annotation sources to build annogroups for.
 #   "all"            - every source that has a parser plugin AND data available
 #   [ pfam, gene3d ] - an explicit subset (each must have parsers/<source>.py)
-# Parser plugins present: pfam, panther (positional -> 4 types) and go (whole-
-# protein -> 3 types). "all" auto-includes every parser, so new parsers need no
-# edit here.
+# Parser plugins present (12): positional (4 types) pfam, panther, gene3d, cdd,
+# smart, superfamily, funfam, tmbed, metapredict; whole-protein (3 types) go,
+# deeploc, signalp. "all" auto-includes every parser, so new parsers need no edit
+# here. A subset is a list, e.g. [ pfam, go, tmbed ].
 sources: "all"
 
 # =============================================================================

gigantic_project-COPYME/subprojects/annogroups/BLOCK_build_annogroups/workflow-COPYME-build_annogroups/ai/scripts/parsers/cdd.py

@@ -0,0 +1,116 @@
+# AI: Claude Code | Opus 4.8 (1M context) | 2026 June 28 | Purpose: annogroups source parser for CDD (per-species InterProScan-parsed cdd files)
+# Human: Eric Edsinger
+
+"""
+annogroups source parser — cdd.
+
+Reads the per-species CDD annotation files exposed by annotations_hmms at
+  annotations_hmms/output_to_input/BLOCK_interproscan_parsed/cdd/cdd-<phyloname>.tsv
+and returns, per sequence, the list of CDD features (all positional).
+
+NCBI Conserved Domain Database domains (cd accessions). Each match carries residue coordinates (Match_Start / Match_End), so
+cdd participates in all four annogroup types (feature, combination,
+architecture, absent) -- combination groups by the whole-protein set of CDD
+accessions, architecture by their sub-protein N->C arrangement.
+
+The parsed cdd files share the identical 15-column self-documenting schema as
+the pfam files, so this parser mirrors parsers/pfam.py (only SOURCE and the source
+subpath differ). Example accession: cd05751.
+
+Parser-plugin contract (see utils_annogroups): expose SOURCE and
+parse_source_features( workflow_root, config ) -> { sequence_id: [ Feature, ... ] }.
+"""
+
+import sys
+from pathlib import Path
+
+sys.path.insert( 0, str( Path( __file__ ).parent.parent ) )
+import utils_annogroups as U
+
+SOURCE = "cdd"
+
+# Path to this source's annotation files, relative to the workflow root. The config
+# exposes the annotations_hmms output_to_input root; each parser appends its own
+# source-specific subpath.
+RELATIVE_SUBPATH = "BLOCK_interproscan_parsed/cdd"
+
+
+def parse_source_features( workflow_root: Path, config: dict ) -> dict:
+    """sequence_identifier -> [ Feature(accession, start, stop, is_positional=True) ]."""
+    annotations_dir = U.resolve_input_path(
+        workflow_root, config[ "inputs" ][ "annotations_hmms_dir" ]
+    ) / RELATIVE_SUBPATH
+
+    if not annotations_dir.is_dir():
+        print( f"CRITICAL ERROR: cdd annotation directory not found: {annotations_dir}", file = sys.stderr )
+        print( "  Verify annotations_hmms exposed BLOCK_interproscan_parsed/cdd/ in output_to_input.", file = sys.stderr )
+        sys.exit( 1 )
+
+    source_files = sorted( annotations_dir.glob( "cdd-*.tsv" ) )
+    if not source_files:
+        print( f"CRITICAL ERROR: no cdd-*.tsv files in {annotations_dir}", file = sys.stderr )
+        sys.exit( 1 )
+
+    proteins___features = {}
+    for source_file in source_files:
+        with open( source_file, 'r' ) as input_source:
+            # Protein_Identifier (...)\tMD5 (...)\tSequence_Length (...)\tAnalysis_Database (...)\tAccession (...)\tDescription (...)\tMatch_Start (...)\tMatch_End (...)\t...
+            header_ids___indices = U.build_header_index( input_source.readline() )
+            index_protein = header_ids___indices[ "Protein_Identifier" ]
+            index_accession = header_ids___indices[ "Accession" ]
+            index_start = header_ids___indices[ "Match_Start" ]
+            index_stop = header_ids___indices[ "Match_End" ]
+            for line in input_source:
+                line = line.rstrip( '\n' )
+                if not line:
+                    continue
+                parts = line.split( '\t' )
+                protein = parts[ index_protein ]
+                accession = parts[ index_accession ]
+                try:
+                    start = int( parts[ index_start ] )
+                    stop = int( parts[ index_stop ] )
+                except ( ValueError, IndexError ):
+                    # A positional match must have integer coordinates; a malformed
+                    # row is a genuine upstream data problem -- fail fast (section 36).
+                    print( f"CRITICAL ERROR: non-integer Match_Start/Match_End in {source_file.name}: {line[ :120 ]}", file = sys.stderr )
+                    sys.exit( 1 )
+                proteins___features.setdefault( protein, [] ).append(
+                    U.Feature( accession = accession, start = start, stop = stop, is_positional = True )
+                )
+
+    return proteins___features
+
+
+def parse_source_definitions( workflow_root: Path, config: dict ) -> dict:
+    """
+    accession -> human-readable definition (CDD signature description, column
+    'Description'). First non-empty description seen per accession wins.
+
+    Optional half of the parser contract: a source that exposes this lets the builder
+    attach Annotation_Definitions to the annogroup map. Sources without descriptions
+    can omit this function (the builder then emits empty definitions).
+    """
+    annotations_dir = U.resolve_input_path(
+        workflow_root, config[ "inputs" ][ "annotations_hmms_dir" ]
+    ) / RELATIVE_SUBPATH
+
+    source_files = sorted( annotations_dir.glob( "cdd-*.tsv" ) )
+
+    accessions___definitions = {}
+    for source_file in source_files:
+        with open( source_file, 'r' ) as input_source:
+            header_ids___indices = U.build_header_index( input_source.readline() )
+            index_accession = header_ids___indices[ "Accession" ]
+            index_description = header_ids___indices[ "Description" ]
+            for line in input_source:
+                line = line.rstrip( '\n' )
+                if not line:
+                    continue
+                parts = line.split( '\t' )
+                accession = parts[ index_accession ]
+                description = U.sanitize_annotation_text( parts[ index_description ] ) if index_description < len( parts ) else ''
+                if description and description != '-' and accession not in accessions___definitions:
+                    accessions___definitions[ accession ] = description
+
+    return accessions___definitions