Skip to content

Commit 6186551

Browse files
committed
research_notebook: consolidate to single project-root location only
Per discussion 2026-05-26: there is ONE research_notebook/ per project, at gigantic_project-COPYME/research_notebook/. Per-subproject research_notebook/ directories are forbidden — they were inconsistent in practice (phylonames had only a README, genomesDB had loose audit TSVs at root, trees_species had reversed-word-order subdirs user_research/ai_research, secretome had none) and the chat-capture system always wrote to the project-root location anyway. Convention updates (gigantic_conventions.md): - §1: rewritten to mandate single project-root research_notebook/, forbid per-subproject ones, and document the optional subproject-<name>/ scoping pattern inside research_user/ and research_ai/ for per-subproject scratch. - §25: clarified that "research_user ships empty / research_ai ships with docs" applies only to the project-root location. Disk changes: - Moved genomesDB's 4 audit files (ncbi_md5_verification, ncbi_status_sweep, source_provenance_audit .md + .tsv) from subprojects/genomesDB/research_notebook/ → research_notebook/research_user/subproject-genomesDB/ (sandbox content gitignored — these were never in version control) - Deleted per-subproject research_notebook/ from phylonames + genomesDB + trees_species (trees_species had only zero-byte .gitkeep files in reversed-word-order user_research/ and ai_research/ dirs; phylonames had only a docs README) Doc updates across 11 files: - phylonames + genomesDB + trees_species subproject README and AI_GUIDE directory trees no longer list a subproject-level research_notebook/; replaced with inline note pointing at project-root sandbox. - genomesDB STEP_1-sources README + AI_GUIDE: rewrote sections that incorrectly implied per-STEP research_notebook/ existence. - genomesDB STEP_1 workflow README + ai/AI_GUIDE: fixed broken relative paths (was ../../research_notebook/... which from workflow level pointed at STEP level; corrected to ../../../../research_notebook/... reaching project root). - References to bare research_notebook/research_user/ left as-is where unambiguous — §1 mandates the single project-root location, so the bare path is unambiguously that. Note: chat-capture filename format and TRANSCRIPT_CAPTURE_LOG.md schema unchanged. The PreCompact hook and Save Chat! script were already writing only to the project-root location.
1 parent 3303916 commit 6186551

13 files changed

Lines changed: 98 additions & 118 deletions

File tree

gigantic_project-COPYME/ai/ai_FYIs/gigantic_conventions.md

Lines changed: 37 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,25 +8,49 @@ At the end of cleanup, this is the source for writing the top-level `GIGANTIC/`
88

99
---
1010

11-
## 1. `research_notebook/` is a sandbox, outside GIGANTIC conventions
11+
## 1. `research_notebook/` is a sandbox, outside GIGANTIC conventions — and there is ONE, at project root only
1212

13-
`research_notebook/` directories at any level (project, subproject, BLOCK/STEP) are
14-
personal sandboxes. They have:
13+
There is **exactly one** `research_notebook/` per project, at the
14+
project root: `gigantic_project-COPYME/research_notebook/`. Per-subproject
15+
`research_notebook/` directories are forbidden. (Earlier templates
16+
allowed per-subproject and per-BLOCK/STEP sandboxes; that pattern was
17+
consolidated — the per-subproject dirs were inconsistent in practice
18+
and the chat-capture system always wrote to the project-root location
19+
anyway.)
20+
21+
The project-root sandbox has:
1522

1623
- **No structure requirements** — any files, any naming, any subdirectory layout
1724
- **No naming conventions** — users organize however they like
1825

19-
The only hard rule: **GIGANTIC must never pull from anything inside a
20-
`research_notebook/`.** Workflows, scripts, and pipelines treat these directories as
26+
The only hard rule: **GIGANTIC must never pull from anything inside
27+
`research_notebook/`.** Workflows, scripts, and pipelines treat it as
2128
invisible.
2229

23-
If a user wants GIGANTIC to use a file that lives in their `research_notebook/`,
24-
they symlink it into `gigantic_project-COPYME/INPUT_user/` (the canonical
25-
project-level input directory). GIGANTIC reads from `INPUT_user/`, never from
30+
If a user wants GIGANTIC to use a file that lives in
31+
`research_notebook/`, they symlink it into
32+
`gigantic_project-COPYME/INPUT_user/` (the canonical project-level
33+
input directory). GIGANTIC reads from `INPUT_user/`, never from
2634
`research_notebook/`.
2735

28-
This separation is what lets `research_notebook/` stay completely free-form while
29-
GIGANTIC's reproducibility guarantees remain intact.
36+
This separation is what lets `research_notebook/` stay completely
37+
free-form while GIGANTIC's reproducibility guarantees remain intact.
38+
39+
**Per-subproject scoping** (when useful): organize inside the project-root
40+
sandbox with `subproject-<name>/` subdirs, e.g.:
41+
42+
```
43+
research_notebook/
44+
├── research_user/
45+
│ ├── subproject-genomesDB/ # user's per-subproject scratch
46+
│ ├── subproject-phylonames/
47+
│ └── species70/ # or any user-defined organization
48+
└── research_ai/
49+
├── sessions/ # chat captures (per §9)
50+
└── subproject-phylonames/ # AI-side per-subproject scratch (when needed)
51+
```
52+
53+
But this is a convenience pattern, not a requirement — the sandbox is free-form.
3054

3155
---
3256

@@ -510,7 +534,9 @@ All four are in `.gitignore`. When clearing a stuck run, deleting these
510534

511535
## 25. `research_user/` ships as empty directory; `research_ai/` ships with documentation
512536

513-
Refines §1:
537+
Refines §1 (and applies only to the single project-root
538+
`research_notebook/` — per-subproject `research_notebook/` directories
539+
are forbidden):
514540

515541
- `research_notebook/research_user/` is the user's wild-west sandbox. The
516542
template ships only the empty directory (`.gitkeep` only — no README,

gigantic_project-COPYME/subprojects/genomesDB/AI_GUIDE.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -267,7 +267,9 @@ genomesDB/
267267
│ ├── STEP_3-databases/
268268
│ └── STEP_4-create_final_species_set/
269269
270-
├── research_notebook/ # Personal workspace + AI session captures (§1, §25)
270+
│ (no per-subproject research_notebook/ — single project-root sandbox at
271+
│ gigantic_project-COPYME/research_notebook/ per §1, §25; chat captures
272+
│ land at research_notebook/research_ai/sessions/ per §9)
271273
272274
├── STEP_1-sources/
273275
│ ├── README.md

gigantic_project-COPYME/subprojects/genomesDB/README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -219,7 +219,9 @@ genomesDB/
219219
│ ├── STEP_3-databases/
220220
│ └── STEP_4-create_final_species_set/ # The final species set canonically lives here
221221
222-
├── research_notebook/ # Personal workspace + AI session captures (§1, §25)
222+
│ (no per-subproject research_notebook/ — single project-root sandbox at
223+
│ gigantic_project-COPYME/research_notebook/ per §1, §25; chat captures
224+
│ land at research_notebook/research_ai/sessions/ per §9)
223225
224226
├── STEP_1-sources/
225227
│ ├── README.md

gigantic_project-COPYME/subprojects/genomesDB/STEP_1-sources/AI_GUIDE.md

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -47,12 +47,12 @@ Unlike other GIGANTIC subprojects that automatically download or generate data,
4747
- Users may have custom or unpublished genomes
4848
- Complete control over what enters GIGANTIC
4949

50-
**research_notebook/research_user/ directory**:
51-
- Located at `research_notebook/research_user/`
52-
- User's personal workspace for source data
53-
- NOT part of GIGANTIC (only README is version-controlled)
54-
- Can contain anything: raw downloads, scripts, notes, analyses
55-
- Workflow reads FROM here but doesn't manage it
50+
**User sandbox for source data** (per §1, §25):
51+
- Single project-root location: `gigantic_project-COPYME/research_notebook/research_user/`
52+
- (No per-subproject `research_notebook/` exists.)
53+
- Wild-west — any files, any organization
54+
- NOT part of GIGANTIC (contents are gitignored; never version-controlled)
55+
- Workflow reads FROM here (via INPUT_user symlinks per §17, §18) but doesn't manage it
5656

5757
---
5858

@@ -189,7 +189,7 @@ Workflow run logs are saved to each workflow's `ai/logs/` directory. AI sessions
189189
| `workflow-*/INPUT_user/source_manifest.tsv` | 4-column manifest | **YES** (required) |
190190
| `workflow-*/START_HERE-user_config.yaml` | Project name, options | **YES** (project name) |
191191
| `output_to_input/STEP_1-sources/T1_proteomes/` | Symlinks for STEP_2 | No (auto-created) |
192-
| `research_notebook/research_user/` | User's personal source data | Personal space |
192+
| `../../../research_notebook/research_user/` (project-root sandbox per §1, §25) | User's personal source data | Personal space (gitignored) |
193193

194194
---
195195

@@ -265,11 +265,11 @@ STEP_1-sources/
265265
├── README.md # Human-readable overview
266266
├── AI_GUIDE.md # THIS FILE
267267
├── output_to_input/ # Symlinks passed to STEP_2
268-
│ └── proteomes/ # Created by workflow
268+
│ └── STEP_1-sources/T1_proteomes/ # Created by workflow
269269
│ ├── Species1.aa -> ...
270270
│ └── proteome_manifest.tsv
271-
├── research_notebook/research_user/ # User's personal workspace
272-
└── README.md # Only this is part of GIGANTIC
271+
│ (no per-STEP research_notebook/ — single project-root sandbox at
272+
gigantic_project-COPYME/research_notebook/ per §1, §25)
273273
└── workflow-COPYME-ingest_source_data/
274274
├── README.md
275275
├── RUN-workflow.sh

gigantic_project-COPYME/subprojects/genomesDB/STEP_1-sources/README.md

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -132,20 +132,24 @@ See `workflow-COPYME-ingest_source_data/README.md` for detailed instructions.
132132

133133
---
134134

135-
## research_notebook/research_user/ Directory
135+
## Personal workspace for source data
136136

137-
The `research_notebook/research_user/` directory is the user's personal workspace (per gigantic_conventions §1, §25):
138-
- Store source genomes/proteomes before ingestion
139-
- Keep custom scripts, analyses, notes
140-
- Organize however you want
137+
Per conventions §1 + §25, the user's wild-west sandbox is the single
138+
project-root location `gigantic_project-COPYME/research_notebook/research_user/`
139+
(no per-subproject `research_notebook/` exists). Use it for:
140+
- Source genomes/proteomes before ingestion
141+
- Custom scripts, analyses, notes
142+
- Anything else, organized however you want
141143

142-
**NOT part of GIGANTIC** - only the README is version-controlled.
144+
The sandbox is gitignored — content is NOT version-controlled.
143145

144146
---
145147

146-
## Research Notebook
148+
## Run logs vs AI session logs
147149

148-
Workflow run logs are saved to each workflow's `ai/logs/` directory. AI sessions are extracted project-wide to `research_notebook/research_ai/sessions/`.
150+
Workflow run logs are saved to each workflow's `ai/logs/` directory.
151+
AI chat session transcripts (per §9) are captured project-wide at
152+
`gigantic_project-COPYME/research_notebook/research_ai/sessions/`.
149153

150154
---
151155

@@ -162,9 +166,9 @@ STEP_1-sources/
162166
├── README.md # This file
163167
├── AI_GUIDE.md # Guide for AI assistants
164168
├── output_to_input/ # Symlinks passed to STEP_2
165-
│ └── proteomes/ # Created by workflow
166-
├── research_notebook/research_user/ # Your personal workspace (not part of GIGANTIC)
167-
└── README.md # Only this README is in GIGANTIC
169+
│ └── STEP_1-sources/T1_proteomes/ # Created by workflow
170+
│ (no per-STEP research_notebook/ — single project-root sandbox at
171+
gigantic_project-COPYME/research_notebook/ per §1, §25)
168172
└── workflow-COPYME-ingest_source_data/ # The ingestion workflow template
169173
├── README.md
170174
├── RUN-workflow.sh

gigantic_project-COPYME/subprojects/genomesDB/STEP_1-sources/workflow-COPYME-ingest_source_data/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Ingest user-provided genome, proteome, and annotation files into GIGANTIC for do
2121

2222
## Quick Start
2323

24-
1. Place your source files in the project-level `INPUT_user/genomic_resources/` subdirectories (genomes/, proteomes/, annotations/) or somewhere else accessible (e.g., `../../research_notebook/research_user/`)
24+
1. Place your source files in the project-level `INPUT_user/genomic_resources/` subdirectories (genomes/, proteomes/, annotations/) or somewhere else accessible (e.g., the project-root sandbox at `../../../../research_notebook/research_user/` per §1, §25)
2525

2626
2. **Ensure files follow GIGANTIC naming convention**:
2727
```

gigantic_project-COPYME/subprojects/genomesDB/STEP_1-sources/workflow-COPYME-ingest_source_data/ai/AI_GUIDE.md

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,7 @@ Users must have data files in the project-level `INPUT_user/genomic_resources/`
136136
- `../../../../INPUT_user/genomic_resources/proteomes/` - Project-level proteome files (.aa)
137137
- `../../../../INPUT_user/genomic_resources/annotations/` - Project-level annotation files (.gff3/.gtf)
138138
- `../../../../INPUT_user/genomic_resources/maps/` - Identifier mapping files (.tsv)
139-
- `../../research_notebook/research_user/` - Personal research data (alternative)
139+
- `../../../../research_notebook/research_user/` - Project-root sandbox (alternative, per §1, §25)
140140
- Any accessible path on the system
141141

142142
### Step 2: Ensure Files Follow GIGANTIC Naming Convention
@@ -367,11 +367,17 @@ bash ai/scripts/003_ai-bash-create_output_symlinks.sh \
367367

368368
---
369369

370-
## What's NOT in research_notebook/research_user/
370+
## On the user sandbox (research_notebook/research_user/)
371371

372-
**Important**: The `research_notebook/research_user/` directory at the STEP_1 level is:
373-
- NOT part of GIGANTIC (only the README is)
374-
- User's personal space for source data
372+
Per conventions §1, §25, the user sandbox lives at the **single
373+
project-root location**: `gigantic_project-COPYME/research_notebook/research_user/`
374+
(there is no per-STEP or per-subproject `research_notebook/`). Properties:
375+
376+
- NOT part of GIGANTIC (contents are gitignored; never version-controlled)
377+
- User's personal space for any source data, scripts, notes
378+
- Wild-west organization — no naming/structure rules
375379
- Can contain anything the user wants
376380

377-
The workflow reads FROM user_research (or anywhere else) but stores GIGANTIC-managed copies in OUTPUT_pipeline.
381+
The workflow reads from `INPUT_user/` (which contains symlinks into the
382+
project-root sandbox per §17, §18), not directly from the sandbox itself.
383+
GIGANTIC-managed copies of ingested data live in `OUTPUT_pipeline/`.

gigantic_project-COPYME/subprojects/genomesDB/research_notebook/.gitkeep

Whitespace-only changes.

gigantic_project-COPYME/subprojects/phylonames/AI_GUIDE.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,9 @@ phylonames/
7171
│ └── maps/ # Convenience symlink to most recent STEP
7272
│ └── [project]_map-genus_species_X_phylonames.tsv (symlink)
7373
74-
├── research_notebook/ # Personal workspace + AI session captures (§1, §25)
74+
│ (no per-subproject research_notebook/ — single project-root sandbox at
75+
│ gigantic_project-COPYME/research_notebook/ per §1, §25; chat captures
76+
│ land at research_notebook/research_ai/sessions/ per §9)
7577
│ ├── research_user/ # User sandbox (ships empty)
7678
│ └── research_ai/sessions/ # Captured chat transcripts (§9)
7779

gigantic_project-COPYME/subprojects/phylonames/README.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -204,9 +204,9 @@ phylonames/
204204
│ └── maps/ # Convenience landing dir; the .tsv inside is a symlink
205205
│ └── [project]_map-genus_species_X_phylonames.tsv (symlink, updated to whichever STEP ran last)
206206
207-
├── research_notebook/ # Sandbox + AI session captures (§1, §9, §25)
208-
├── research_user/ # User-controlled (ships empty)
209-
└── research_ai/sessions/ # Captured chat transcripts (.jsonl.gz)
207+
│ (no per-subproject research_notebook/ — the single project-root one
208+
at gigantic_project-COPYME/research_notebook/ serves all subprojects;
209+
see conventions §1, §9, §25)
210210
211211
├── STEP_1-generate_and_evaluate/
212212
│ ├── AI_GUIDE.md # STEP-level AI guide
@@ -260,7 +260,9 @@ phylonames/
260260
**AI lab-notebook logs**: Each workflow run creates a timestamped log in
261261
its own `ai/logs/` directory. This documents what the workflow did, when,
262262
with what inputs, and what it produced. Captured AI chat transcripts live
263-
separately in `research_notebook/research_ai/sessions/` (project-wide).
263+
in the single project-root sandbox at
264+
`gigantic_project-COPYME/research_notebook/research_ai/sessions/` (see
265+
conventions §1, §9, §25).
264266

265267
---
266268

@@ -344,7 +346,8 @@ If you have custom phylonames to apply:
344346

345347
1. **Stage your overrides via the project-level INPUT_user arena**
346348
(canonical pattern). Put the real file in your sandbox under
347-
`research_notebook/research_user/`, then symlink it into the
349+
`gigantic_project-COPYME/research_notebook/research_user/` (the
350+
single project-root sandbox per §1), then symlink it into the
348351
project-level `INPUT_user/phylonames/` slot:
349352
```bash
350353
cd ../../INPUT_user/phylonames

0 commit comments

Comments
 (0)