Skip to content

Commit 2690186

Browse files
000specificcursoragent
000specific
and
cursoragent
committed
resolve_groups: Leonid updates — 4-structure deconvolution and annotated composite detail tables
Add Script 006 annotation index, restrict deconvolution to structures 001/003/031/032, and join PFAM/PANTHER/GO/Gene_Families/Gene_Groups columns onto all composite-clades detail tables. COPYME-only; RUN instances stay gitignored per GIGANTIC convention. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 2087100 commit 2690186

15 files changed

Lines changed: 1294 additions & 237 deletions

‎gigantic_project-COPYME/subprojects/sequence_groups_X_species/BLOCK_resolve_groups/AI_GUIDE.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# AI_GUIDE — BLOCK_resolve_groups (sequence_groups_X_species)
2+
3+
<!-- ============================================================================
4+
AI: Claude Code | Opus 4.8 | 2026 July 02
5+
Human: Eric Edsinger
6+
============================================================================ -->
7+
8+
**For AI Assistants**: Read `../AI_GUIDE-sequence_groups_X_species.md` first for the
9+
subproject overview and concepts. This guide covers the one block that runs the
10+
resolver workflow.
11+
12+
## Where this fits
13+
14+
- **UP**: subproject guide [`../AI_GUIDE-sequence_groups_X_species.md`](../AI_GUIDE-sequence_groups_X_species.md); subproject [`../README.md`](../README.md); project [`../../../AI_GUIDE.md`](../../../AI_GUIDE.md)
15+
- **DOWN**: workflow template [`workflow-COPYME-resolve_groups/`](workflow-COPYME-resolve_groups/) and its workflow guide [`workflow-COPYME-resolve_groups/ai/AI_GUIDE-resolve_groups_workflow.md`](workflow-COPYME-resolve_groups/ai/AI_GUIDE-resolve_groups_workflow.md)
16+
- **IN**: producer group sets from `../../{orthogroups,annogroups,trees_gene_families,trees_gene_groups}/output_to_input/`; species-tree clades from `../../trees_species/output_to_input/`
17+
- **OUT**: `../output_to_input/<producer>/<group_set_label>/` (downstream) and `../upload_to_server/` (data server), both via the interface layer (§2, §38)
18+
19+
## What this block does
20+
21+
`sequence_groups_X_species` has **one block**: it resolves a sequence-group set onto
22+
the species-tree clades. A single producer-agnostic NextFlow workflow reads one group
23+
set (via a Script 001 producer adapter) and overlays its membership onto the species
24+
tree three ways:
25+
26+
| Script | Output | What |
27+
|--------|--------|------|
28+
| 001 adapt_sequence_group_membership | 1-output | producer → standard membership (`SequenceGroup_ID, Sequence_Identifier, Genus_Species`) |
29+
| 002 species_tree_deconvolution | 2-output | member sequence + species counts per clade (union; scope via `deconvolution_structures`, default all 105) |
30+
| 003 per_species_sequence_map | 3-output | member sequence identifiers per species |
31+
| 004 composite_clades | 4-output | four algorithms + 242 detail tables (with PFAM/PANTHER/GO/Gene_Families/Gene_Groups annotation columns when `annotation_index` is set) |
32+
| 006 build_annotation_index | annotation_index/ | cross-producer sequence → annotation index (runs once; gates Script 004 detail tables) |
33+
34+
An optional per-group `group_attributes` table is carried, opaque, onto the per-group
35+
rows (annogroups supplies its `annogroup_map.tsv`, adding Source / Annogroup_Type /
36+
Annotation_Definitions / …). See the workflow guide for the mechanism.
37+
38+
## Runs
39+
40+
Each run resolves the producers listed in `START_HERE-user_config.yaml` (currently
41+
seven group sets: orthogroups; annogroups pfam / go / panther; gene_families;
42+
gene_groups hugo_hgnc + snap_family). Copy `workflow-COPYME-resolve_groups/` to a new
43+
`workflow-RUN_N-resolve_groups/`, edit config if needed, then `bash RUN-workflow.sh`.
44+
45+
Leonid (2026-07) updates in COPYME: `deconvolution_structures` (001/003/031/032 only),
46+
`annotation_index` + Script 006 (PFAM/PANTHER/GO/Gene_Families/Gene_Groups columns on
47+
all 242 composite-clades detail tables per producer). **RUN_4** is the first run from
48+
this template.

‎gigantic_project-COPYME/subprojects/sequence_groups_X_species/BLOCK_resolve_groups/workflow-COPYME-resolve_groups/INPUT_user/composite_clades_manifest.tsv‎

Lines changed: 227 additions & 50 deletions
Large diffs are not rendered by default.

‎gigantic_project-COPYME/subprojects/sequence_groups_X_species/BLOCK_resolve_groups/workflow-COPYME-resolve_groups/RUN-workflow.sh‎

Lines changed: 62 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,18 @@
77
################################################################################
88
#
99
# PURPOSE:
10-
# Resolve ONE sequence-group set (orthogroups, annogroups, gene families, ...)
11-
# onto the species-tree clades: standard membership (001), deconvolution (002),
12-
# per-species sequence map (003), composite clades (004).
10+
# Resolve MANY sequence-group sets (orthogroups, annogroups pfam/go/panther,
11+
# gene families, gene groups, ...) onto the species-tree clades in ONE run:
12+
# 001 standard membership, 002 deconvolution (4-structure scope), 003 per-species map,
13+
# 006 annotation index (once), 004 composite clades (242 detail tables w/ annotations).
1314
#
1415
# USAGE: bash RUN-workflow.sh
1516
#
1617
# BEFORE RUNNING, edit START_HERE-user_config.yaml:
17-
# - group_set_label (e.g. "species70_X_OrthoHMM")
18-
# - species_set_name, producer (e.g. "orthogroups")
19-
# - inputs.producer_membership / clade_species_mappings / composite_clades_manifest
20-
# - composite_clades block (building-block clade groups)
18+
# - producers: (membership paths + optional group_attributes per producer)
19+
# - deconvolution_structures (default 001/003/031/032; [] = all 105)
20+
# - annotation_index (paths for Script 006; required for annotated detail tables)
21+
# - inputs.clade_species_mappings / composite_clades_manifest
2122
# - execution_mode ("local" or "slurm"); for slurm set slurm_account / slurm_qos
2223
#
2324
# OUTPUT:
@@ -41,9 +42,9 @@ read_config() {
4142
}
4243

4344
EXECUTION_MODE=$(read_config "execution_mode" "local")
44-
GROUP_SET_LABEL=$(read_config "group_set_label" "")
45-
PRODUCER=$(read_config "producer" "")
4645
SPECIES_SET=$(read_config "species_set_name" "")
46+
# Multi-producer: ONE run resolves every entry under `producers:` in the config.
47+
PRODUCER_COUNT=$(grep -c '^[[:space:]]*- producer:' START_HERE-user_config.yaml)
4748

4849
# Workflow directory name (used below to build output_to_input symlink targets).
4950
WORKFLOW_DIR_NAME="$(basename "${SCRIPT_DIR}")"
@@ -88,7 +89,7 @@ echo ""
8889

8990
# ---- validate prerequisites ------------------------------------------------
9091
[ -f "START_HERE-user_config.yaml" ] || { echo "ERROR: START_HERE-user_config.yaml not found"; exit 1; }
91-
echo "Configuration: group_set=${GROUP_SET_LABEL} producer=${PRODUCER} species_set=${SPECIES_SET}"
92+
echo "Configuration: ${PRODUCER_COUNT} producer(s) species_set=${SPECIES_SET}"
9293
echo ""
9394

9495
# ---- run NextFlow ----------------------------------------------------------
@@ -103,53 +104,64 @@ if [ $EXIT_CODE -ne 0 ]; then
103104
exit $EXIT_CODE
104105
fi
105106

106-
# ---- output_to_input symlinks ---------------------------------------------
107-
# Real files in OUTPUT_pipeline/{2,3,4}-output/; symlinks under
108-
# ../../output_to_input/<group_set_label>/ for downstream consumers.
107+
# ---- output_to_input symlinks (per producer) ------------------------------
108+
# Real files live under <subproject>/<block>/<workflow>/OUTPUT_pipeline/<label>/{2,3,4}-output/;
109+
# downstream consumers read symlinks under the SUBPROJECT-root output_to_input/
110+
# in a per-producer namespace: output_to_input/<producer>/<group_set_label>/ so the
111+
# many group sets stay navigable (annogroups/, orthogroups/, gene_families/, gene_groups/).
109112
echo ""
110-
echo "Creating output_to_input symlinks for ${GROUP_SET_LABEL}..."
111-
# output_to_input lives at the SUBPROJECT root; real files live under
112-
# <subproject>/<block>/<workflow>/OUTPUT_pipeline/. So a symlink at
113-
# output_to_input/<label>/ reaches them via ../../<block>/<workflow>/...
113+
echo "Creating output_to_input symlinks for ${PRODUCER_COUNT} producer(s)..."
114114
BLOCK_DIR_NAME="$(basename "$(dirname "${SCRIPT_DIR}")")"
115-
# Per-producer namespace: output_to_input/<producer>/<group_set_label>/ so the many
116-
# group sets stay navigable (annogroups/, orthogroups/, gene_families/, gene_groups/).
117-
SHARED_DIR="../../output_to_input/${PRODUCER}/${GROUP_SET_LABEL}"
118-
# replace stale state for this group set (output_to_input holds only symlinks)
119-
find "${SHARED_DIR}" -mindepth 1 -maxdepth 1 -name '*.tsv' -type l -delete 2>/dev/null
120-
rm -rf "${SHARED_DIR}/composite_clades_detail_tables" 2>/dev/null
121-
mkdir -p "${SHARED_DIR}"
122-
SYMLINK_COUNT=0
123-
link_outputs() {
124-
# link every .tsv in OUTPUT_pipeline/$1 into SHARED_DIR (flat)
125-
for f in OUTPUT_pipeline/$1/*.tsv; do
126-
[ -f "$f" ] || continue
127-
ln -sf "../../../${BLOCK_DIR_NAME}/${WORKFLOW_DIR_NAME}/$f" "${SHARED_DIR}/$(basename "$f")"
128-
SYMLINK_COUNT=$((SYMLINK_COUNT+1))
129-
done
130-
}
131-
link_outputs "2-output" # deconvolution union tables
132-
link_outputs "3-output" # per-species map
133-
link_outputs "4-output" # composite per-group + summary
134-
# composite detail tables (a subdir)
135-
if [ -d "OUTPUT_pipeline/4-output/composite_clades_detail_tables" ]; then
136-
mkdir -p "${SHARED_DIR}/composite_clades_detail_tables"
137-
for f in OUTPUT_pipeline/4-output/composite_clades_detail_tables/*.tsv; do
138-
[ -f "$f" ] || continue
139-
ln -sf "../../../../${BLOCK_DIR_NAME}/${WORKFLOW_DIR_NAME}/$f" "${SHARED_DIR}/composite_clades_detail_tables/$(basename "$f")"
140-
SYMLINK_COUNT=$((SYMLINK_COUNT+1))
141-
done
115+
116+
# (producer, group_set_label) pairs, parsed from the config with the workflow's Python.
117+
PRODUCER_SPECS="$(python3 -c "
118+
import yaml
119+
with open( 'START_HERE-user_config.yaml' ) as handle:
120+
config = yaml.safe_load( handle )
121+
for producer in config.get( 'producers', [] ):
122+
print( producer[ 'producer' ] + '\t' + producer[ 'group_set_label' ] )
123+
")"
124+
if [ -z "${PRODUCER_SPECS}" ]; then
125+
echo "ERROR: could not parse any producers from START_HERE-user_config.yaml"; exit 1
142126
fi
143-
echo " output_to_input/${GROUP_SET_LABEL}/ -> ${SYMLINK_COUNT} symlinks"
127+
128+
while IFS=$'\t' read -r PRODUCER GROUP_SET_LABEL; do
129+
[ -n "${GROUP_SET_LABEL}" ] || continue
130+
OUT_LABEL_DIR="OUTPUT_pipeline/${GROUP_SET_LABEL}"
131+
SHARED_DIR="../../output_to_input/${PRODUCER}/${GROUP_SET_LABEL}"
132+
# replace stale state for this group set (output_to_input holds only symlinks)
133+
find "${SHARED_DIR}" -mindepth 1 -maxdepth 1 -name '*.tsv' -type l -delete 2>/dev/null
134+
rm -rf "${SHARED_DIR}/composite_clades_detail_tables" 2>/dev/null
135+
mkdir -p "${SHARED_DIR}"
136+
SYMLINK_COUNT=0
137+
for sub in 2-output 3-output 4-output; do
138+
for f in "${OUT_LABEL_DIR}/${sub}"/*.tsv; do
139+
[ -f "$f" ] || continue
140+
ln -sf "../../../${BLOCK_DIR_NAME}/${WORKFLOW_DIR_NAME}/$f" "${SHARED_DIR}/$(basename "$f")"
141+
SYMLINK_COUNT=$((SYMLINK_COUNT+1))
142+
done
143+
done
144+
# composite detail tables (a subdir)
145+
if [ -d "${OUT_LABEL_DIR}/4-output/composite_clades_detail_tables" ]; then
146+
mkdir -p "${SHARED_DIR}/composite_clades_detail_tables"
147+
for f in "${OUT_LABEL_DIR}/4-output/composite_clades_detail_tables"/*.tsv; do
148+
[ -f "$f" ] || continue
149+
ln -sf "../../../../${BLOCK_DIR_NAME}/${WORKFLOW_DIR_NAME}/$f" "${SHARED_DIR}/composite_clades_detail_tables/$(basename "$f")"
150+
SYMLINK_COUNT=$((SYMLINK_COUNT+1))
151+
done
152+
fi
153+
echo " output_to_input/${PRODUCER}/${GROUP_SET_LABEL}/ -> ${SYMLINK_COUNT} symlinks"
154+
done <<< "${PRODUCER_SPECS}"
144155

145156
echo ""
146157
echo "========================================================================"
147158
echo "SUCCESS! sequence_groups_X_species resolve_groups complete."
148-
echo " OUTPUT_pipeline/1-output/ standard membership"
149-
echo " OUTPUT_pipeline/2-output/ deconvolution (sequence + species counts per clade)"
150-
echo " OUTPUT_pipeline/3-output/ per-species sequence map"
151-
echo " OUTPUT_pipeline/4-output/ composite clades"
152-
echo " output_to_input/${GROUP_SET_LABEL}/"
159+
echo " Per producer, under OUTPUT_pipeline/<group_set_label>/:"
160+
echo " 1-output/ standard membership"
161+
echo " 2-output/ deconvolution (sequence + species counts per clade)"
162+
echo " 3-output/ per-species sequence map"
163+
echo " 4-output/ composite clades"
164+
echo " Downstream symlinks: output_to_input/<producer>/<group_set_label>/"
153165
echo "Completed: $(date)"
154166
echo "========================================================================"
155167

0 commit comments

Comments
 (0)