phylonames: full §38/§3/§29 compliance refactor + conventions §40/§41

Convention additions:
- §38 refined: explicit "subproject-level only — never per-STEP, per-BLOCK,
  per-workflow-RUN"; user-defined upload_to_server structure allowed via
  manifest dest_name overrides
- §2 refined: output_to_input internal structure mandatorily mirrors
  producer paths (STEP_<N>/, BLOCK_<name>/, unit/) for provenance
- §40 NEW: bidirectional discoverability — producer AI_GUIDEs list their
  consumers ("downstream consumers" section)
- §41 NEW: three subproject types (STEP / BLOCK / UNIT) — internal
  organization is subproject-natural; interface layer is standardized

Phylonames refactor (§38 compliance):
- DELETE legacy flat publisher RUN-update_upload_to_server.sh
  (subproject-level + STEP_1 duplicate)
- DELETE per-STEP upload_to_server/ dirs (STEP_1 + STEP_2) — Eric's
  local data inside was untracked
- DELETE STEP_1 duplicate RUN-clean_and_record_subproject.sh (per-STEP
  copy of the subproject-level script; one path-adjusted line different)
- NEW subproject-level RUN-update_upload_to_server.sh: thin wrapper
  around the canonical shared helper at
  gigantic_project-COPYME/server/ai/update_upload_to_server.py
  (matches trees_gene_families pattern)
- NEW upload_manifest.tsv at each workflow-COPYME-*/ level
  (STEP_1: publish taxonomy_summary html/md from 4-output;
   STEP_2: publish updated taxonomy_summary html/md from 2-output)

Phylonames doc cleanup:
- Rename 4 AI_GUIDE-X.md → AI_GUIDE.md (§3 compliance):
  AI_GUIDE-phylonames.md, STEP_1/AI_GUIDE-generate_and_evaluate.md,
  STEP_1/workflow-COPYME/ai/AI_GUIDE-phylonames_workflow.md,
  STEP_2/AI_GUIDE-apply_user_phylonames.md
- NEW STEP_2/workflow-COPYME-apply_user_phylonames/ai/AI_GUIDE.md
  (was missing; pairs with the new workflow README)
- Bulk-fix cross-references in all phylonames docs: AI_GUIDE-*.md →
  AI_GUIDE.md; ../../AI_GUIDE-project.md (deleted) → ../../AI_GUIDE.md
- Rewrite both workflow READMEs (STEP_1 + STEP_2) to remove
  RUN-workflow.sbatch references; document unified driver per §29;
  add publishing-to-server section
- Update subproject README.md: directory tree reflects current state
  (no per-STEP upload_to_server; manifests at workflow-COPYME-* level;
  drop user_research/ in favor of research_notebook/research_user/);
  "Sharing Data via GIGANTIC Server" section rewritten for new
  subproject-level publisher
- Update subproject AI_GUIDE.md: same directory tree fix; drop
  RUN-workflow.sbatch from Key Files table; add Downstream Consumers
  section per §40 listing every subproject that consumes phylonames
  output_to_input/maps/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 000specific

gigantic_project-COPYME/ai/ai_FYIs/gigantic_conventions.md

@@ -39,9 +39,11 @@ specific outputs with downstream subprojects:
   (lowercase `o`)
 - Entries inside `output_to_input/` are **symlinks** pointing into the canonical
   `OUTPUT_pipeline/` of the appropriate `workflow-RUN_N-*` directory
-- The directory structure inside `output_to_input/` (typically organized by BLOCK
-  or STEP name) keeps sequential runs from overwriting each other and preserves
-  clear provenance
+- The directory structure inside `output_to_input/` is organized by
+  `STEP_<N>-<name>/` or `BLOCK_<name>/` subdirectories matching the
+  producer's structure. This keeps sequential runs from overwriting each
+  other and preserves clear provenance (consumer sees which step generated
+  what)
 
 Downstream subprojects always read from `<upstream_subproject>/output_to_input/`,
 never from inside an `OUTPUT_pipeline/` directly.
@@ -829,17 +831,43 @@ Full documentation lives in `server/README.md` (user-facing) and
 Each subproject exposes selected outputs to the project's data server
 through a fixed interface:
 
-- **`<subproject>/upload_to_server/`** holds the published view of that
-  subproject's data
-- Entries inside are **symlinks** pointing into the canonical
-  `OUTPUT_pipeline/` of the appropriate `workflow-RUN_N-*` directory
-  (per §35; the run-instance dirs hold the actual data)
-- A **`upload_manifest.tsv`** at `<workflow-RUN_*>/upload_manifest.tsv`
-  controls which outputs publish (one manifest per canonical RUN dir;
-  see §39)
-- Each subproject has its own **`RUN-update_upload_to_server.sh`** that
-  invokes the shared helper `gigantic_project-COPYME/server/ai/update_upload_to_server.py`
-  to create/refresh the symlinks in `upload_to_server/`
+- **`<subproject>/upload_to_server/`** is the single publish destination
+  per subproject (subproject-level; **never per-STEP, per-BLOCK, or
+  per-workflow-RUN** — consumers and the server look in exactly one
+  place per subproject). The current state of any subproject with
+  `<STEP>/upload_to_server/` dirs is a legacy deviation to be refactored.
+- Internal structure inside `upload_to_server/` is **user-facing** —
+  organized for collaborators to browse the project, not for GIGANTIC's
+  internal bookkeeping. The shared helper's default is to mirror
+  producer paths:
+  ```
+  <subproject>/upload_to_server/
+    └── STEP_<N>-<name>/         (or BLOCK_<name>/, or unit-name/)
+        └── workflow-RUN_<K>-<name>/   (canonical RUN per §39)
+            └── N-output/
+                └── <file>       (symlink to the actual output)
+  ```
+  This default works well when the natural collaborator-facing framework
+  IS the producer's structure. Per-manifest `dest_name` overrides can
+  reshape paths into a user-defined framework when the natural framework
+  for collaborators differs from the internal pipeline layout (e.g.,
+  grouping by species set, by hypothesis, by deliverable type rather
+  than by STEP).
+
+  Note distinction from `output_to_input/` (§2): `output_to_input/`
+  structure **mirrors producer paths** mandatorily (preserves provenance
+  for downstream subprojects); `upload_to_server/` structure is for
+  human collaborators and **can deviate** from producer paths when a
+  different organization serves the user better.
+- Each canonical `workflow-RUN_*/` carries its own
+  **`upload_manifest.tsv`** controlling which of its outputs publish
+  (one manifest per canonical RUN dir; see §39)
+- Each subproject has a **`RUN-update_upload_to_server.sh`** at
+  subproject level that invokes the shared helper at
+  `gigantic_project-COPYME/server/ai/update_upload_to_server.py`. The
+  shared helper walks the subproject, finds every `upload_manifest.tsv`
+  inside any `workflow-RUN_*/`, and assembles the nested
+  `upload_to_server/` tree above.
 - The server reads `upload_to_server/` symlinks transparently — **there
   is no copy or sync step**. Follow-the-symlink at HTTP request time.
 
@@ -896,6 +924,74 @@ workflow" section).
 
 ---
 
+## 40. Bidirectional discoverability: producer AI_GUIDEs list their consumers
+
+When a subproject, STEP, BLOCK, or workflow **consumes** data from
+another subproject's `output_to_input/`, update the **producer's**
+relevant AI_GUIDE to note the consumer. This bidirectional cross-
+reference makes the data-flow graph navigable from either end.
+
+**Example**: trees_species' STEP that produces species-tree structures
+gets a note in its AI_GUIDE saying "downstream consumers: orthogroups_X_ocl
+reads these structures; annotations_X_ocl reads them via orthogroups_X_ocl's
+output_to_input/."
+
+**Why**: an AI walking into a subproject and finding it produces data
+should be able to answer "who uses this?" without grep'ing the whole
+tree. The producer-side reference solves this directly. Without it, you
+need a separate dependency map (drifts) or a cross-tree search (slow,
+incomplete).
+
+**How**: add a "Downstream consumers" section to the producer's relevant
+AI_GUIDE (the one closest to the output_to_input/ source) when a
+consumer relationship is established. Keep it short:
+
+```markdown
+## Downstream consumers
+
+The `output_to_input/<subdir>/` exposed here is read by:
+
+- **<consumer subproject>** — `<consumer subproject>/...` consumes
+  `<which file(s) and what for>`. See `<consumer subproject>/AI_GUIDE.md`
+  for the consumer side.
+```
+
+Apply when surfacing a new producer-consumer pair during cleanup or
+during new development.
+
+---
+
+## 41. Subproject internal organization: three natural types (STEP, BLOCK, UNIT)
+
+GIGANTIC subprojects come in three structural shapes, chosen to fit the
+scientific use case. Internal structure is **subproject-natural** — do
+not force a single shape on all subprojects. (Compare with the
+**interface layer** — `upload_to_server/`, `output_to_input/`,
+`RUN-update_upload_to_server.sh` at subproject root — which IS
+standardized per §2, §38, etc., regardless of internal type.)
+
+| Type | When to use | Internal layout | Examples |
+|------|-------------|-----------------|----------|
+| **STEP** | Sequential pipeline; STEP_<N+1> depends on STEP_<N> | `<subproject>/STEP_<N>-<descriptor>/workflow-COPYME-<name>/` | `phylonames` (STEP_1-generate_and_evaluate → STEP_2-apply_user_phylonames); `genomesDB` (STEP_1-sources → ... → STEP_4-create_final_species_set) |
+| **BLOCK** | Parallel/alternative tools, methods, or analyses producing comparable outputs | `<subproject>/BLOCK_<descriptor>/workflow-COPYME-<name>/` | `orthogroups` (BLOCK_orthohmm, BLOCK_orthofinder, BLOCK_broccoli — pick your tool); `annotations_hmms` (BLOCK_interproscan, BLOCK_deeploc, BLOCK_signalp — different annotations); `hotspots` (BLOCK_self_blast, BLOCK_identify_hotspots) |
+| **UNIT** | Per-unit instances of a templated workflow chain (each unit goes through the same STEP_1 → STEP_2 → ... pipeline) | `<subproject>/<unit_prefix>_COPYME/STEP_<N>/workflow-COPYME-<name>/` (template) and `<subproject>/<unit_prefix>-<name>/STEP_<N>/workflow-COPYME-<name>/` (per-unit instances) | `trees_gene_families` (gene_family_COPYME template + gene_family-<name> instances); `trees_gene_groups` (similar) |
+
+**Mixed or hybrid types**: a subproject occasionally needs more than one
+type (e.g., a STEP-organized subproject where one STEP has BLOCK
+alternatives inside it). That's fine — apply the type that fits at each
+level. Don't invent custom types.
+
+**Naming**:
+- `STEP_<N>-<descriptor>` — zero-padded number with descriptive suffix
+- `BLOCK_<descriptor>` — no number; BLOCKs are alternatives, not ordered
+- `<unit_prefix>_COPYME` for the unit template (e.g., `gene_family_COPYME`)
+- `<unit_prefix>-<unit_name>` for unit instances (e.g.,
+  `gene_family-innexin_pannexin_channels`)
+- `workflow-COPYME-<descriptor>` for the workflow template inside any
+  STEP/BLOCK/unit (§35)
+
+---
+
 <!-- Add new conventions below as they surface during per-directory review. -->
 <!-- User shorthand "gcon" = "please add this to gigantic_conventions.md". -->
 

…ojects/phylonames/AI_GUIDE-phylonames.md …OPYME/subprojects/phylonames/AI_GUIDE.mdgigantic_project-COPYME/subprojects/phylonames/AI_GUIDE-phylonames.md renamed to gigantic_project-COPYME/subprojects/phylonames/AI_GUIDE.md gigantic_project-COPYME/subprojects/phylonames/AI_GUIDE-phylonames.md renamed to gigantic_project-COPYME/subprojects/phylonames/AI_GUIDE.md

@@ -1,6 +1,6 @@
 # AI Guide: phylonames Subproject
 
-**For AI Assistants**: Read `../../AI_GUIDE-project.md` first for GIGANTIC overview, directory structure, and general patterns. This guide covers phylonames-specific concepts and troubleshooting.
+**For AI Assistants**: Read `../../../AI_GUIDE.md` (project root) first for GIGANTIC overview, directory structure, and general patterns. This guide covers phylonames-specific concepts and troubleshooting.
 
 **Location**: `gigantic_project-*/subprojects/phylonames/`
 
@@ -21,11 +21,11 @@
 
 | User needs... | Go to... |
 |---------------|----------|
-| GIGANTIC overview, directory structure | `../../AI_GUIDE-project.md` |
+| GIGANTIC overview, directory structure | `../../../AI_GUIDE.md` (project root) |
 | Phylonames concepts, troubleshooting | This file |
-| STEP_1 overview (generate and evaluate) | `STEP_1-generate_and_evaluate/AI_GUIDE-generate_and_evaluate.md` |
-| STEP_1 workflow execution | `STEP_1-generate_and_evaluate/workflow-COPYME-generate_phylonames/ai/AI_GUIDE-phylonames_workflow.md` |
-| STEP_2 overview (apply user phylonames) | `STEP_2-apply_user_phylonames/AI_GUIDE-apply_user_phylonames.md` |
+| STEP_1 overview (generate and evaluate) | `STEP_1-generate_and_evaluate/AI_GUIDE.md` |
+| STEP_1 workflow execution | `STEP_1-generate_and_evaluate/workflow-COPYME-generate_phylonames/ai/AI_GUIDE.md` |
+| STEP_2 overview (apply user phylonames) | `STEP_2-apply_user_phylonames/AI_GUIDE.md` |
 | STEP_2 workflow execution | `STEP_2-apply_user_phylonames/workflow-COPYME-apply_user_phylonames/ai/` |
 
 ---
@@ -53,39 +53,44 @@
 
 ```
 phylonames/
-├── README.md                        # Human documentation
-├── AI_GUIDE-phylonames.md           # THIS FILE
+├── README.md                                   # User-facing documentation
+├── AI_GUIDE.md                                 # THIS FILE
 │
-├── user_research/                   # Personal workspace for this subproject
-├── upload_to_server/                # Server sharing
+├── RUN-clean_and_record_subproject.sh          # Subproject-level cleanup + session recording
+├── RUN-update_upload_to_server.sh              # Subproject-level publisher (thin wrapper around shared helper, §38)
 │
-├── output_to_input/                                # Outputs for downstream subprojects
+├── upload_to_server/                           # Single publish destination per §38
+│                                               # (auto-populated by RUN-update_upload_to_server.sh; no per-STEP upload_to_server)
+│
+├── output_to_input/                            # Outputs for downstream subprojects (§2)
 │   ├── STEP_1-generate_and_evaluate/
 │   │   └── maps/
-│   │       └── [project]_map-genus_species_X_phylonames.tsv
+│   │       └── [project]_map-genus_species_X_phylonames.tsv  (symlink)
 │   ├── STEP_2-apply_user_phylonames/
 │   │   └── maps/
-│   │       └── [project]_map-genus_species_X_phylonames.tsv
-│   └── maps/                                       # Convenience symlink to whichever STEP ran last
-│       └── [project]_map-genus_species_X_phylonames.tsv  # SYMLINK
+│   │       └── [project]_map-genus_species_X_phylonames.tsv  (symlink)
+│   └── maps/                                   # Convenience symlink to most recent STEP
+│       └── [project]_map-genus_species_X_phylonames.tsv      (symlink)
+│
+├── research_notebook/                          # Personal workspace + AI session captures (§1, §25)
+│   ├── research_user/                          # User sandbox (ships empty)
+│   └── research_ai/sessions/                   # Captured chat transcripts (§9)
 │
 ├── STEP_1-generate_and_evaluate/
-│   ├── AI_GUIDE-generate_and_evaluate.md     # STEP-level guide
-│   ├── RUN-clean_and_record_subproject.sh    # Cleanup + AI session recording
-│   ├── RUN-update_upload_to_server.sh        # Update server symlinks
-│   ├── upload_to_server/                     # Taxonomy summaries for server
-│   │   └── taxonomy_summaries/
+│   ├── AI_GUIDE.md                             # STEP-level guide
 │   │
 │   └── workflow-COPYME-generate_phylonames/
-│       ├── RUN-workflow.sh                   # bash RUN-workflow.sh
-│       ├── RUN-workflow.sbatch               # sbatch RUN-workflow.sbatch
-│       ├── START_HERE-user_config.yaml            # User edits project name here
-│       ├── INPUT_user/                       # Species list (copied from project INPUT_user at runtime)
-│       ├── OUTPUT_pipeline/                  # Results
-│       └── ai/                               # Nextflow pipeline and scripts
-│           ├── AI_GUIDE-phylonames_workflow.md
+│       ├── README.md                           # User-facing workflow quickstart
+│       ├── RUN-workflow.sh                     # Unified driver (§29: local or slurm via execution_mode)
+│       ├── START_HERE-user_config.yaml         # Config: project name, execution_mode, slurm.*
+│       ├── upload_manifest.tsv                 # Server publish manifest (§38, §39)
+│       ├── INPUT_user/                         # Species list (auto-copied from project default if absent)
+│       ├── OUTPUT_pipeline/                    # Results (1-output through 5-output)
+│       └── ai/
+│           ├── AI_GUIDE.md
 │           ├── main.nf
 │           ├── nextflow.config
+│           ├── conda_environment.yml
 │           └── scripts/
 │               ├── 001_ai-bash-download_ncbi_taxonomy.sh
 │               ├── 002_ai-python-generate_phylonames.py
@@ -94,23 +99,30 @@ phylonames/
 │               └── 005_ai-python-write_run_log.py
 │
 └── STEP_2-apply_user_phylonames/
-    ├── AI_GUIDE-apply_user_phylonames.md     # STEP-level guide
+    ├── AI_GUIDE.md                             # STEP-level guide
     │
     └── workflow-COPYME-apply_user_phylonames/
-        ├── RUN-workflow.sh                   # bash RUN-workflow.sh
-        ├── RUN-workflow.sbatch               # sbatch RUN-workflow.sbatch
-        ├── START_HERE-user_config.yaml            # User edits project name and user phylonames path
-        ├── INPUT_user/                       # User phylonames TSV goes here
-        ├── OUTPUT_pipeline/                  # Results
-        └── ai/                               # Nextflow pipeline and scripts
+        ├── README.md                           # User-facing workflow quickstart
+        ├── RUN-workflow.sh                     # Unified driver (§29)
+        ├── START_HERE-user_config.yaml         # Config: project name, user_phylonames path, etc.
+        ├── upload_manifest.tsv                 # Server publish manifest
+        ├── INPUT_user/                         # User phylonames TSV
+        ├── OUTPUT_pipeline/                    # Results (1-output through 3-output)
+        └── ai/
+            ├── AI_GUIDE.md
             ├── main.nf
             ├── nextflow.config
+            ├── conda_environment.yml
             └── scripts/
                 ├── 001_ai-python-apply_user_phylonames.py
                 ├── 002_ai-python-generate_taxonomy_summary.py
                 └── 003_ai-python-write_run_log.py
 ```
 
+Per §38 and §41, phylonames is a STEP-organized subproject and has ONE
+subproject-level `upload_to_server/` (no per-STEP `upload_to_server/` —
+those were deleted in the 2026-05-26 cleanup).
+
 ---
 
 ## Phylonames-Specific Concepts
@@ -273,24 +285,51 @@ workflow-*/ai/logs/
 |------|---------|-------------|
 | `../../INPUT_user/species_set/species_list.txt` | Project-wide default species list | **YES** |
 | `STEP_1-*/workflow-*/START_HERE-user_config.yaml` | STEP_1 project name and options | **YES** |
-| `STEP_1-*/workflow-*/RUN-workflow.sbatch` | STEP_1 SLURM account/qos | **YES** (SLURM) |
+| `STEP_1-*/workflow-*/START_HERE-user_config.yaml` | STEP_1 project name, execution_mode, slurm.account/qos | **YES** |
 | `STEP_1-*/workflow-*/INPUT_user/species_list.txt` | STEP_1 species list override (optional, auto-copied from project default) | **YES** (to override) |
-| `STEP_2-*/workflow-*/START_HERE-user_config.yaml` | STEP_2 project name, user phylonames path, unofficial marking | **YES** |
-| `STEP_2-*/workflow-*/RUN-workflow.sbatch` | STEP_2 SLURM account/qos | **YES** (SLURM) |
+| `STEP_2-*/workflow-*/START_HERE-user_config.yaml` | STEP_2 project name, user phylonames path, unofficial marking, execution_mode, slurm.account/qos | **YES** |
 | `STEP_2-*/workflow-*/INPUT_user/user_phylonames.tsv` | User-provided phyloname overrides | **YES** |
 | `output_to_input/maps/*.tsv` | Output for downstream subprojects (symlink to latest STEP) | No |
 | `output_to_input/STEP_1-*/maps/*.tsv` | STEP_1 mapping output | No |
 | `output_to_input/STEP_2-*/maps/*.tsv` | STEP_2 mapping output (with user overrides) | No |
 
 ---
 
+## Downstream consumers (per §40)
+
+The `output_to_input/maps/[project]_map-genus_species_X_phylonames.tsv`
+exposed here is consumed by virtually every downstream GIGANTIC
+subproject, because phylonames are the canonical species identifiers
+used throughout the framework:
+
+- **genomesDB** — uses phylonames to name standardized proteome files
+  and to label species across the database
+- **orthogroups** — labels orthogroup memberships by phyloname
+- **annotations_hmms** — keys per-species annotation files by phyloname
+- **trees_species** — labels species in candidate species trees by
+  phyloname
+- **trees_gene_families**, **trees_gene_groups** — label gene-family /
+  gene-group tree tips by phyloname
+- **orthogroups_X_ocl**, **annotations_X_ocl** — propagate phyloname
+  labels through OCL analyses
+- **gene_sizes**, **homolog_counts**, **hotspots**, **secretome**,
+  **one_direction_homologs**, and so on — all use phylonames
+
+In practice every subproject that operates per-species reads from
+`phylonames/output_to_input/maps/` at some point. **phylonames must run
+first** in any GIGANTIC project (see top of this file).
+
 ## Next Steps After phylonames
 
 Guide users to:
-1. **Review STEP_1 output** - Check the taxonomy summary for numbered clades or misclassifications
-2. **Run STEP_2 if needed** - If any species need corrected phylonames, create `user_phylonames.tsv` and run STEP_2
-3. **genomesDB** - Set up proteome database using phylonames for file naming
-4. **Keep the mapping** - All downstream subprojects reference `output_to_input/maps/`
+1. **Review STEP_1 output** — check the taxonomy summary for numbered
+   clades or misclassifications
+2. **Run STEP_2 if needed** — if any species need corrected phylonames,
+   create `user_phylonames.tsv` and run STEP_2
+3. **genomesDB next** — set up proteome database using phylonames for
+   file naming (the canonical second subproject to run)
+4. **Keep the mapping intact** — all downstream subprojects reference
+   `output_to_input/maps/`
 
 ---