Skip to content

Commit f3d4de9

Browse files
000specificclaude
000specific
and
claude
committed
genomesDB: §38/§3/§29 compliance refactor (Batches A-C + partial D)
Batches A-C (structural/publisher compliance): - Rename 9 AI_GUIDE-X.md → AI_GUIDE.md (§3 compliance): subproject + 4 STEPs (sources, standardize_and_evaluate, databases, create_final_species_set) + 4 workflows - Bulk cross-ref fix across all genomesDB docs (AI_GUIDE-*.md → AI_GUIDE.md; AI_GUIDE-project.md → AI_GUIDE.md) - DELETE 8 per-STEP duplicate RUN scripts (4 RUN-update_upload_to_server.sh + 4 RUN-clean_and_record_subproject.sh) - DELETE 2 per-STEP upload_to_server/ dirs (STEP_3 + STEP_4) - DELETE legacy flat publisher RUN-update_upload_to_server.sh - DELETE legacy upload_to_server/{upload_manifest.tsv, README.md} (legacy flat-publisher format with all entries commented out) - NEW subproject-level RUN-update_upload_to_server.sh: thin wrapper around shared helper (mirrors phylonames + trees_gene_families) - NEW upload_manifest.tsv at each of 4 workflow-COPYME-*/ levels with per-STEP-appropriate publish defaults: - STEP_1: minimal (validation/log entries, all "no" by default) - STEP_2: quality outputs prominent (BUSCO, gfastats, summary — "yes") - STEP_3: BLAST DB index summary "yes"; raw .phr/.pin/.psq files "no" - STEP_4: final species manifest + selection decisions "yes" - Restore upload_to_server/.gitkeep to preserve the empty dir Batch D (content updates — partial; STEP_0 decision pending): - Bulk sed: sbatch refs → unified RUN-workflow.sh per §29 mention; user_research/ → research_notebook/research_user/ per §1/§25 - Subproject README + AI_GUIDE: rewrote directory-tree sections to reflect actual current structure (no per-STEP upload_to_server; no per-STEP duplicate RUN scripts; upload_manifest.tsv at workflow level; research_notebook/ properly shown) - Subproject AI_GUIDE: added §40 "Downstream consumers" section listing every subproject that consumes from genomesDB/output_to_input/ DEFERRED (Batch D continuation): - STEP_0-prepare_proteomes references throughout README + AI_GUIDE (directory doesn't exist on disk; ~20 lines of stale documentation to resolve — decision needed: drop the references entirely, or retain as "future addition" with clear "NOT YET IMPLEMENTED" marker) - STEP-level AI_GUIDE freshening pass (mostly done via sed, some structural review still useful) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 059d051 commit f3d4de9

35 files changed

Lines changed: 283 additions & 3435 deletions

File tree

gigantic_project-COPYME/subprojects/genomesDB/AI_GUIDE-genomesDB.md renamed to gigantic_project-COPYME/subprojects/genomesDB/AI_GUIDE.md

Lines changed: 75 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# AI Guide: genomesDB Subproject
22

3-
**For AI Assistants**: Read `../../AI_GUIDE-project.md` first for GIGANTIC overview, directory structure, and general patterns. This guide covers genomesDB-specific concepts and the four-step architecture.
3+
**For AI Assistants**: Read `../../AI_GUIDE.md` first for GIGANTIC overview, directory structure, and general patterns. This guide covers genomesDB-specific concepts and the four-step architecture.
44

55
**Location**: `gigantic_project-COPYME/subprojects/genomesDB/`
66

@@ -23,7 +23,7 @@
2323

2424
| User needs... | Go to... |
2525
|---------------|----------|
26-
| GIGANTIC overview, directory structure | `../../AI_GUIDE-project.md` |
26+
| GIGANTIC overview, directory structure | `../../AI_GUIDE.md` |
2727
| genomesDB concepts, pipeline structure | This file |
2828
| STEP_0 prepare_proteomes workflow | `STEP_0-prepare_proteomes/workflow-COPYME-*/ai/AI_GUIDE-*_workflow.md` |
2929
| STEP_1 sources workflow | `STEP_1-sources/workflow-COPYME-*/ai/AI_GUIDE-*_workflow.md` |
@@ -233,70 +233,69 @@ Homo_sapiens-genome_ncbi_GCF_000001405.40-downloaded_20240115.aa
233233

234234
```
235235
genomesDB/
236-
├── README.md # Human documentation
237-
├── AI_GUIDE-genomesDB.md # THIS FILE
238-
├── RUN-clean_and_record_subproject.sh # Cleanup for entire subproject
239-
├── RUN-update_upload_to_server.sh # Update server symlinks
236+
├── README.md # User-facing documentation
237+
├── AI_GUIDE.md # THIS FILE
240238
│
241-
├── output_to_input/ # Final outputs for downstream
242-
├── upload_to_server/ # Server sharing
239+
├── RUN-clean_and_record_subproject.sh # Subproject-level cleanup + session recording
240+
├── RUN-update_upload_to_server.sh # Subproject-level publisher (§38; thin wrapper around shared helper)
243241
│
244-
├── STEP_0-prepare_proteomes/ # (OPTIONAL) Prepare T1 from evigene transcriptomes
245-
│ ├── README.md
246-
│ ├── output_to_input/ # T1 proteomes for STEP_1
247-
│ └── workflow-COPYME-prepare_proteomes/
248-
│ ├── INPUT_user/
249-
│ ├── OUTPUT_pipeline/
250-
│ ├── RUN-workflow.sh
251-
│ ├── RUN-workflow.sbatch
252-
│ └── ai/
242+
├── upload_to_server/ # Single publish destination (§38)
243+
│
244+
├── output_to_input/ # Outputs for downstream subprojects (§2)
245+
│ ├── STEP_1-sources/
246+
│ ├── STEP_2-standardize_and_evaluate/
247+
│ ├── STEP_3-databases/
248+
│ └── STEP_4-create_final_species_set/
249+
│
250+
├── research_notebook/ # Personal workspace + AI session captures (§1, §25)
253251
│
254252
├── STEP_1-sources/
255253
│ ├── README.md
256-
│ ├── AI_GUIDE-sources.md
257-
│ ├── RUN-clean_and_record_subproject.sh
254+
│ ├── AI_GUIDE.md
258255
│ └── workflow-COPYME-ingest_source_data/
256+
│ ├── README.md
257+
│ ├── RUN-workflow.sh # Unified driver (§29)
258+
│ ├── START_HERE-user_config.yaml
259+
│ ├── upload_manifest.tsv # Server publish manifest (§38, §39)
259260
│ ├── INPUT_user/
260-
│ │ └── source_manifest.tsv # User creates this
261+
│ │ └── source_manifest.tsv # User creates this
261262
│ ├── OUTPUT_pipeline/
262-
│ ├── RUN-workflow.sh # Local execution
263-
│ ├── RUN-workflow.sbatch # SLURM execution
264-
│ └── ai/
263+
│ └── ai/ # NextFlow pipeline + scripts
265264
│
266265
├── STEP_2-standardize_and_evaluate/
267266
│ ├── README.md
268-
│ ├── AI_GUIDE-standardize_and_evaluate.md
269-
│ ├── RUN-clean_and_record_subproject.sh
267+
│ ├── AI_GUIDE.md
270268
│ └── workflow-COPYME-standardize_evaluate_build_gigantic_genomesdb/
271-
│ ├── INPUT_user/
272-
│ ├── OUTPUT_pipeline/
273-
│ ├── RUN-workflow.sh # Local execution
274-
│ ├── RUN-workflow.sbatch # SLURM execution
269+
│ ├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
270+
│ ├── upload_manifest.tsv
271+
│ ├── INPUT_user/, OUTPUT_pipeline/
275272
│ └── ai/
276273
│
277274
├── STEP_3-databases/
278275
│ ├── README.md
279-
│ ├── AI_GUIDE-databases.md
280-
│ ├── RUN-clean_and_record_subproject.sh
276+
│ ├── AI_GUIDE.md
281277
│ └── workflow-COPYME-build_gigantic_genomesDB/
282-
│ ├── INPUT_user/
283-
│ ├── OUTPUT_pipeline/
284-
│ ├── RUN-workflow.sh # Local execution
285-
│ ├── RUN-workflow.sbatch # SLURM execution
278+
│ ├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
279+
│ ├── upload_manifest.tsv
280+
│ ├── INPUT_user/, OUTPUT_pipeline/
286281
│ └── ai/
287282
│
288283
└── STEP_4-create_final_species_set/
289284
├── README.md
290-
├── AI_GUIDE-create_final_species_set.md
291-
├── RUN-clean_and_record_subproject.sh
285+
├── AI_GUIDE.md
292286
└── workflow-COPYME-create_final_species_set/
293-
├── INPUT_user/
294-
├── OUTPUT_pipeline/
295-
├── RUN-workflow.sh # Local execution
296-
├── RUN-workflow.sbatch # SLURM execution
287+
├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
288+
├── upload_manifest.tsv
289+
├── INPUT_user/, OUTPUT_pipeline/
297290
└── ai/
298291
```
299292

293+
Per §38 + §41, genomesDB is a STEP-organized subproject with ONE
294+
subproject-level `upload_to_server/` and one subproject-level
295+
`output_to_input/` (no per-STEP duplicates of either; deleted in the
296+
2026-05-26 cleanup). Per §29, unified `RUN-workflow.sh` drives both
297+
local and SLURM execution via the YAML `execution_mode` key.
298+
300299
---
301300

302301
## Data Flow Between Steps
@@ -400,7 +399,7 @@ ls output_to_input/STEP_4-create_final_species_set/
400399
| `STEP_4-create_final_species_set/workflow-*/START_HERE-user_config.yaml` | Paths to STEP_2/STEP_3 outputs | **YES** |
401400
| `STEP_4-create_final_species_set/workflow-*/INPUT_user/selected_species.txt` | Species selection (optional) | **YES** (optional) |
402401
| `output_to_input/STEP_4-create_final_species_set/` | Final species set | No |
403-
| `upload_to_server/upload_manifest.tsv` | What to share | **YES** |
402+
| `STEP_*/workflow-COPYME-*/upload_manifest.tsv` | Per-workflow publish manifest (§38) | **YES** (to customize what publishes) |
404403

405404
---
406405

@@ -420,9 +419,40 @@ ls output_to_input/STEP_4-create_final_species_set/
420419

421420
---
422421

422+
## Downstream consumers (per §40)
423+
424+
genomesDB feeds essentially every downstream subproject that operates on
425+
standardized proteomes / genomes / annotations:
426+
427+
- **annotations_hmms** — reads standardized proteomes (`.aa` files) from
428+
`genomesDB/output_to_input/STEP_2-standardize_and_evaluate/proteomes/`
429+
to run InterProScan, DeepLoc, SignalP, TMBed, MetaPredict
430+
- **orthogroups** — reads the same standardized proteomes to run
431+
OrthoHMM, OrthoFinder, Broccoli
432+
- **trees_gene_families** — STEP_1 (homolog discovery) reads BLAST
433+
databases from `genomesDB/output_to_input/STEP_3-databases/` for
434+
RBH/RBF searches
435+
- **trees_gene_groups** — same dependency as trees_gene_families
436+
- **trees_species** — uses the species set defined in
437+
`genomesDB/output_to_input/STEP_4-create_final_species_set/` as input
438+
for species-tree topology generation
439+
- **gene_sizes** — reads genomes + annotations from STEP_2 outputs
440+
- **hotspots**, **secretome**, **one_direction_homologs**,
441+
**dark_proteomes**, etc. — all consume standardized proteomes
442+
- **orthogroups_X_ocl**, **annotations_X_ocl** — indirect (through
443+
their producer subprojects)
444+
445+
In practice every "real" GIGANTIC subproject reads from
446+
`genomesDB/output_to_input/` at some point. **genomesDB must run
447+
second** in any GIGANTIC project (after phylonames).
448+
423449
## Next Steps After genomesDB
424450

425451
Guide users to:
426-
1. **annotations_hmms** - Run functional annotations on proteomes
427-
2. **orthogroups** - Identify ortholog groups across species
428-
3. **trees_gene_families** - Build gene family phylogenies
452+
1. **annotations_hmms** — run functional annotations on standardized
453+
proteomes
454+
2. **orthogroups** — identify ortholog groups across species
455+
3. **trees_species** — generate candidate species-tree topologies for
456+
the final species set
457+
4. **trees_gene_families** / **trees_gene_groups** — build per-gene
458+
phylogenies (depend on BLAST databases from STEP_3)

‎gigantic_project-COPYME/subprojects/genomesDB/README.md‎

Lines changed: 41 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -177,72 +177,68 @@ genomesDB/
177177

178178
```
179179
genomesDB/
180-
├── README.md # This file
181-
├── AI_GUIDE-genomesDB.md # AI assistant guidance (subproject level)
182-
├── RUN-clean_and_record_subproject.sh # Cleanup script for entire subproject
183-
├── RUN-update_upload_to_server.sh # Update server sharing symlinks
184-
├── output_to_input/ # Final outputs for downstream subprojects
185-
├── upload_to_server/ # Files to share via GIGANTIC server
180+
├── README.md # This file
181+
├── AI_GUIDE.md # AI assistant guidance (subproject level)
186182
│
187-
├── STEP_0-prepare_proteomes/ # (OPTIONAL) Prepare T1 from evigene transcriptomes
188-
│ ├── README.md
189-
│ ├── output_to_input/ # T1 proteomes for STEP_1
190-
│ └── workflow-COPYME-prepare_proteomes/
191-
│ ├── INPUT_user/
192-
│ ├── OUTPUT_pipeline/
193-
│ ├── RUN-workflow.sh
194-
│ ├── RUN-workflow.sbatch
195-
│ └── ai/
183+
├── RUN-clean_and_record_subproject.sh # Subproject-level cleanup + session recording
184+
├── RUN-update_upload_to_server.sh # Subproject-level publisher (§38)
185+
│
186+
├── upload_to_server/ # Single publish destination per §38
187+
│ # (auto-populated by RUN-update_upload_to_server.sh)
188+
│
189+
├── output_to_input/ # Outputs for downstream subprojects (§2)
190+
│ ├── STEP_1-sources/
191+
│ ├── STEP_2-standardize_and_evaluate/
192+
│ ├── STEP_3-databases/
193+
│ └── STEP_4-create_final_species_set/ # The final species set canonically lives here
194+
│
195+
├── research_notebook/ # Personal workspace + AI session captures (§1, §25)
196196
│
197197
├── STEP_1-sources/
198-
│ ├── README.md # Step-specific documentation
199-
│ ├── AI_GUIDE-sources.md # AI guidance for this step
200-
│ ├── RUN-clean_and_record_subproject.sh # Step-level cleanup
201-
│ ├── output_to_input/ # Outputs passed to STEP_2
198+
│ ├── README.md
199+
│ ├── AI_GUIDE.md # STEP-level guide
202200
│ └── workflow-COPYME-ingest_source_data/
203-
│ ├── INPUT_user/ # Source manifest goes here
204-
│ ├── OUTPUT_pipeline/ # Workflow outputs
205-
│ ├── RUN-workflow.sh # Local execution
206-
│ ├── RUN-workflow.sbatch # SLURM execution
207-
│ └── ai/ # Pipeline scripts
201+
│ ├── README.md
202+
│ ├── RUN-workflow.sh # Unified driver — local or SLURM via execution_mode (§29)
203+
│ ├── START_HERE-user_config.yaml # Config: project name, execution_mode, slurm.*
204+
│ ├── upload_manifest.tsv # Server publish manifest (§38, §39)
205+
│ ├── INPUT_user/ # Source manifest goes here
206+
│ ├── OUTPUT_pipeline/ # Workflow outputs
207+
│ └── ai/ # NextFlow pipeline + scripts
208208
│
209209
├── STEP_2-standardize_and_evaluate/
210210
│ ├── README.md
211-
│ ├── AI_GUIDE-standardize_and_evaluate.md
212-
│ ├── RUN-clean_and_record_subproject.sh
213-
│ ├── output_to_input/ # Outputs passed to STEP_3
211+
│ ├── AI_GUIDE.md
214212
│ └── workflow-COPYME-standardize_evaluate_build_gigantic_genomesdb/
215-
│ ├── INPUT_user/
216-
│ ├── OUTPUT_pipeline/
217-
│ ├── RUN-workflow.sh # Local execution
218-
│ ├── RUN-workflow.sbatch # SLURM execution
213+
│ ├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
214+
│ ├── upload_manifest.tsv # Server publish manifest
215+
│ ├── INPUT_user/, OUTPUT_pipeline/
219216
│ └── ai/
220217
│
221218
├── STEP_3-databases/
222219
│ ├── README.md
223-
│ ├── AI_GUIDE-databases.md
224-
│ ├── RUN-clean_and_record_subproject.sh
225-
│ ├── output_to_input/ # Step outputs (also copied to subproject root)
220+
│ ├── AI_GUIDE.md
226221
│ └── workflow-COPYME-build_gigantic_genomesDB/
227-
│ ├── INPUT_user/
228-
│ ├── OUTPUT_pipeline/
229-
│ ├── RUN-workflow.sh # Local execution
230-
│ ├── RUN-workflow.sbatch # SLURM execution
222+
│ ├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
223+
│ ├── upload_manifest.tsv # Server publish manifest
224+
│ ├── INPUT_user/, OUTPUT_pipeline/
231225
│ └── ai/
232226
│
233227
└── STEP_4-create_final_species_set/
234228
├── README.md
235-
├── AI_GUIDE-create_final_species_set.md
236-
├── RUN-clean_and_record_subproject.sh
237-
├── output_to_input/ # Final species set for downstream subprojects
229+
├── AI_GUIDE.md
238230
└── workflow-COPYME-create_final_species_set/
239-
├── INPUT_user/
240-
├── OUTPUT_pipeline/
241-
├── RUN-workflow.sh # Local execution
242-
├── RUN-workflow.sbatch # SLURM execution
231+
├── README.md, RUN-workflow.sh, START_HERE-user_config.yaml
232+
├── upload_manifest.tsv # Server publish manifest
233+
├── INPUT_user/, OUTPUT_pipeline/
243234
└── ai/
244235
```
245236

237+
Per §38 + §41, genomesDB is a STEP-organized subproject with ONE
238+
subproject-level `upload_to_server/` (no per-STEP `upload_to_server/`).
239+
Per §29, the unified `RUN-workflow.sh` drives both local and SLURM
240+
execution via the YAML `execution_mode` key.
241+
246242
---
247243

248244
## Data Flow

0 commit comments

Comments
 (0)